Directivity control apparatus, directivity control method, storage medium and directivity control system

ABSTRACT

A directivity control apparatus controls a directivity of a sound collected by a first sound collecting unit including a plurality of microphones. The directivity control apparatus includes a directivity forming unit, configured to form a directivity of the sound in a direction toward a monitoring target corresponding to a first designated position in an image displayed on a display unit, and an information obtaining unit, configured to obtain information on a second designated position in the image displayed on the display unit, designated in accordance with a movement of the monitoring target. The directivity forming unit is configured to change the directivity of the sound toward the monitoring target corresponding to the second designated position by referring to the information on the second designated position obtained by the information obtaining unit.

BACKGROUND

1. Field of the Invention

The present invention relates to a directivity control apparatus, adirectivity control method, a storage medium and a directivity controlsystem for controlling directivity of a sound.

2. Description of the Related Art

In the related art, in a monitoring system installed at a predeterminedposition (for example, a ceiling surface) of a factory, a shop (forexample, a retail store or a bank) or a public place (for example, alibrary), one or more camera apparatuses (for example, pan tilt zoom(PTZ) camera apparatuses or omni-directional camera apparatuses) areconnected to the monitoring system through a network to achieve a widefield angle of image data on video within a monitoring target range. Theimage data may include a still image or a moving image.

Since the amount of information obtained in video monitoring is limited,a monitoring system capable of obtaining sound data emitted by aspecific monitoring target (for example, a person) that is presentwithin a field angle of a camera apparatus using a microphone arrayapparatus in which plural microphones are accommodated in addition toone or more camera apparatuses is highly demanded. Further, in themonitoring system, it is necessary to consider a movement of a personwhen the microphone array apparatus collects a sound.

Here, as a related art technique that draws tracking points bydesignating the tracking points from a start point to an end point ofmovement on a TV monitor screen on which an image captured by a TVcamera is projected to make an input operation of a user simple, forexample, a camera platform control apparatus of a TV camera disclosed inJP-A-6-133189 has been proposed.

In camera platform control apparatus of a TV camera disclosed inJP-A-6-133189, an image captured by a TV camera mounted on a cameraplatform provided with pan tilt driver is projected onto a TV monitor,tracking points from a start point to an end point of movement duringautomatic photographing are input on a screen of the TV monitor, and thesequentially input tracking points are sequentially connected to eachother to form a tracking line, tracking data from the start point to theend point of the movement of the tracking line is sequentially read, andautomatic photographing is executed so that the tracking point relatingto the read data is positioned at the center of the photographingscreen. Thus, in the camera platform control apparatus of the TV camera,as the tracking points are input on the screen of the TV monitor, it ispossible to obtain tracking data on pan tilt driving by a simple inputoperation, and to accurately perform a driving control.

SUMMARY

However, since JP-A-6-133189 does not disclose a configuration in whichthe sound emitted by the person projected onto the TV monitor iscollected, for example, even when the configuration disclosed inJP-A-6-133189 is applied to the above-described monitoring system, thesound from the person is not easily collected on the tracking pointsfrom the start point to the end point of the movement and is notcollected with high accuracy.

In order to solve the above-mentioned problems, a non-limited object ofthe present invention is to provide a directivity control apparatus, adirectivity control method, a storage medium and a directivity controlsystem capable of appropriately forming, even when a monitoring targeton an image moves, directivity of a sound with respect to the monitoringtarget in a tracking manner to prevent deterioration monitoring workefficiency of an observer.

A first aspect of the present invention provides a directivity controlapparatus for controlling a directivity of a sound collected by a firstsound collecting unit including a plurality of microphones, thedirectivity control apparatus including: a directivity forming unit,configured to form a directivity of the sound in a direction toward amonitoring target corresponding to a first designated position in animage displayed on a display unit; and an information obtaining unit,configured to obtain information on a second designated position in theimage displayed on the display unit, designated in accordance with amovement of the monitoring target, wherein the directivity forming unitchanges the directivity of the sound toward the monitoring targetcorresponding to the second designated position by referring to theinformation on the second designated position obtained by theinformation obtaining unit.

A second aspect of the present invention provides a directivity controlmethod in a directivity control apparatus for controlling a directivityof a sound collected by a first sound collecting unit including aplurality of microphones, the directivity control method including:forming a directivity of the sound in a direction toward a monitoringtarget corresponding to a first designated position in an imagedisplayed on a display unit; obtaining information on a seconddesignated position in the image displayed on the display unit,designated in accordance with a movement of the monitoring target; andchanging the directivity of the sound toward the monitoring targetcorresponding to the second designated position by referring to theinformation on the second designated position.

A third aspect of the present invention provides a storage medium inwhich a program is stored, the program causing a computer in adirectivity control apparatus for controlling a directivity of a soundcollected by a first sound collecting unit including a plurality ofmicrophones, to execute: forming a directivity of the sound in adirection toward a monitoring target corresponding to a first designatedposition in an image displayed on a display unit; obtaining informationon a second designated position in the image displayed on the displayunit, designated in accordance with a movement of the monitoring target;and changing the directivity of the sound toward the monitoring targetcorresponding to the second designated position by referring to theinformation on the second designated position.

A fourth aspect of the present invention provides a directivity controlsystem including: an imaging unit, configured to capture an image in asound collecting area; a first sound collecting unit including aplurality of microphones, configured to collect a sound in the soundcollecting area; and a directivity control apparatus, configured tocontrol a directivity of the sound collected by the first soundcollecting unit, wherein the directivity control apparatus includes: adirectivity forming unit, configured to form a directivity of the soundin a direction toward a monitoring target corresponding to a firstdesignated position in an image displayed on a display unit; and aninformation obtaining unit, configured to obtain information on a seconddesignated position in the image displayed on the display unit,designated in accordance with a movement of the monitoring target,wherein the directivity forming unit changes the directivity of thesound toward the monitoring target corresponding to the seconddesignated position by referring to the information on the seconddesignated position obtained by the information obtaining unit.

According to any one of the aspects of the present invention, it ispossible to appropriately form, even when a monitoring target in animage moves, directivity of a sound with respect to the monitoringtarget in a tracking manner to prevent deterioration of monitoring workefficiency of an observer.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an operation outline of a directivitycontrol system according to a first embodiment.

FIG. 2 is a block diagram illustrating a first system configurationexample of the directivity control system according to the firstembodiment.

FIG. 3 is a block diagram illustrating a second system configurationexample of the directivity control system according to the firstembodiment.

FIG. 4 is a diagram illustrating an operation example of a manualtracking process.

FIG. 5 is a diagram illustrating an operation example in which when atracking point that is automatically designated in an automatic trackingprocess is wrong, the tracking point is changed by the manual trackingprocess.

FIG. 6 is a diagram illustrating a slow reproduction process in arecording and reproduction mode and a slow reproduction mode.

FIG. 7 is a diagram illustrating an enlarge display process in anenlarge display mode.

FIG. 8A is a diagram illustrating an automatic scroll process after theenlarge display process in the enlarge display mode, FIG. 8B is adiagram illustrating a tracking screen at time t=t1, and FIG. 8C is adiagram illustrating a tracking screen at time t=t2.

FIG. 9A is a flowchart illustrating a first example of an overall flowof the manual tracking process in the directivity control systemaccording to the first embodiment, and FIG. 9B is a flowchartillustrating a second example of the overall flow of the manual trackingprocess in the directivity control system according to the firstembodiment.

FIG. 10A is a flowchart illustrating a first example of an overall flowof the automatic tracking process in the directivity control systemaccording to the first embodiment, and FIG. 10B is a flowchartillustrating the first example of the automatic tracking process shownin FIG. 10A.

FIG. 11A is a flowchart illustrating a second example of the automatictracking process shown in FIG. 10A, and FIG. 11B is a flowchartillustrating an example of a tracking correction process shown in FIG.11A.

FIG. 12 is a flowchart illustrating a third example of the automatictracking process shown in FIG. 10A.

FIG. 13A is a flowchart illustrating an example of a tracking auxiliaryprocess shown in FIG. 9A, and FIG. 13B is a flowchart illustrating anexample of an automatic scroll process shown in FIG. 13A.

FIG. 14A is a flowchart illustrating an example of an automatic scrollprocess necessity determination process shown in FIG. 13B, and FIG. 14Bis a diagram illustrating a scroll necessity determination line in theautomatic scroll process necessity determination process.

FIG. 15A is a flowchart illustrating an example of a tracking connectionprocess shown in FIG. 9A, and FIG. 15B is a flowchart illustrating anexample of a batch connection process shown in FIG. 15A.

FIG. 16A is a diagram illustrating a reproduction start time PT of acollected sound corresponding to a designated position of a user on amovement route between tracking points displayed with respect toone-time movement of a person, and FIG. 16B is a diagram illustrating afirst example of a tracking list.

FIG. 17A is a diagram illustrating a reproduction start time PT of acollected sound corresponding to a designated position of a user on amovement route between different tracking points based on pluralsimultaneous designations, and FIG. 17B is a diagram illustrating asecond example of the tracking list.

FIG. 18A is a diagram illustrating reproduction start times PT and PT′of a collected sound corresponding to respective designated positions ofa user on movement routes between different tracking points based onplural-time designations, and FIG. 18B is a diagram illustrating a thirdexample of the tracking list.

FIG. 19A is a flowchart illustrating an example of an overall flow of amovement route display reproduction process using the tracking list inthe directivity control system according to the first embodiment, FIG.19B is a flowchart illustrating an example of a reproduction start timecalculation process shown in FIG. 19A.

FIG. 20 is a flowchart illustrating an example of a movement routedisplay process shown in FIG. 19A.

FIG. 21A is a flowchart illustrating an example of a sound outputprocess shown in FIG. 9A, and FIG. 21B is a flowchart illustrating anexample of an image privacy protection process shown in FIG. 13A.

FIG. 22A is a diagram illustrating an example of a waveform of a soundsignal corresponding to a pitch before a voice change process, FIG. 22Bis a diagram illustrating an example of a waveform of a sound signalcorresponding to a pitch after the voice change process, and FIG. 22C isa diagram illustrating a shading-off process of an outline of the faceof a detected person.

FIG. 23 is a block diagram illustrating a system configuration exampleof a directivity control system according to a second embodiment.

FIG. 24 is a diagram illustrating an automatic switching process of acamera apparatus used for capturing an image displayed in a displayapparatus.

FIG. 25 is a diagram illustrating an automatic switching process of anomni-directional microphone array apparatus used for collection of asound of a monitoring target.

FIG. 26 is a diagram illustrating a manual switching process of thecamera apparatus used for capturing the image displayed in the displayapparatus.

FIG. 27 is a diagram illustrating a manual switching process of theomni-directional microphone array apparatus used for the soundcollection of the monitoring target.

FIG. 28 is a diagram illustrating a selection process of theomni-directional microphone array apparatus optimal for the soundcollection of the monitoring target.

FIG. 29A is a flowchart illustrating an example of the automaticswitching process of the camera apparatus in the directivity controlsystem according to the second embodiment, FIG. 29B is a flowchartillustrating an example of a camera switching determination processshown in FIG. 29A.

FIG. 30A is a flowchart illustrating an example of the automaticswitching process of the omni-directional microphone array apparatus inthe directivity control system according to the second embodiment, andFIG. 30B is a flowchart illustrating an example of a microphoneswitching determination process shown in FIG. 30A.

FIG. 31A is a flowchart illustrating an example of the manual switchingprocess of the camera apparatus in the directivity control systemaccording to the second embodiment, FIG. 31B is a flowchart illustratingan example of the manual switching process of the omni-directionalmicrophone array apparatus in the directivity control system accordingto the second embodiment.

FIG. 32A is a flowchart illustrating a first example of the selectionprocess of the omni-directional microphone array apparatus optimal inthe directivity control system according to the second embodiment, andFIG. 32B is a flowchart illustrating a second example of the selectionprocess of the omni-directional microphone array apparatus optimal inthe directivity control system according to the second embodiment.

FIG. 33 is a flowchart illustrating a third example of the selectionprocess of the omni-directional microphone array apparatus optimal inthe directivity control system according to the second embodiment.

FIG. 34 is a flowchart illustrating an example of an overall flow of amanual tracking process based on plural simultaneous designations in adirectivity control system according to a modification example of thefirst embodiment.

FIG. 35 is a flowchart illustrating an example of an automatic trackingprocess of plural monitoring targets in the directivity control systemaccording to the modification example of the first embodiment.

FIGS. 36A to 36E are appearance drawings of a housing of theomni-directional microphone array apparatus.

FIG. 37 is a diagram briefly illustrating a delay and sum technique inwhich the omni-directional microphone array apparatus forms directivityof sound data in a direction at an angle θ.

DETAILED DESCRIPTION

Hereinafter, respective embodiments of a directivity control apparatus,a directivity control method, a recording medium and a directivitycontrol system according to the present invention will be described withreference to the accompanying drawings. The directivity control systemin each embodiment is used as a monitoring system (including a mannedmonitoring system and an unmanned monitoring system) installed at afactory, a public facility (for example, a library or an event venue) ora shop (for example, a retail store or a bank).

The present invention may be expressed as a computer-readable recordingmedium on which a program that allows a directivity control apparatusthat is provided as a computer to execute an operation regulated by adirectivity control method or a program that allows a computer toexecute the operation regulated by the directivity control method isrecorded.

First Embodiment

FIG. 1 is a diagram illustrating an operation outline of directivitycontrol systems 100 and 100A according to a first embodiment. FIG. 2 isa block diagram illustrating a first system configuration example of thedirectivity control system 100 according to the first embodiment. FIG. 3is a block diagram illustrating a second system configuration example ofthe directivity control system 100A according to the first embodiment.

Specific configurations of the directivity control systems 100 and 100Awill be described later. First, the operation outline of the directivitycontrol systems 100 and 100A will be briefly described with reference toFIG. 1.

In FIG. 1, a camera apparatus C1 images a monitoring target (forexample, a person HM1) of the directivity control systems 100 and 100Aused as a monitoring system, for example, and transmits data on an imageobtained by the imaging to a directivity control apparatus 3 connectedthrough a network NW.

In the respective embodiments including the present embodiment, theperson HM1 may be stationary or moving, but hereinafter, it is assumedthat the person HM1 is moving. The person HM1 moves from a trackingposition A1 (x1, y1, z0) at a tracking time t1, for example, to atracking position A2 (x2, y2, z0) at a tracking time t2.

Here, a tracking point indicates, when the image of the moving personHM1 obtained by the imaging of the camera apparatus C1 is displayed on atracking screen TRW of a display apparatus 35, a position where a userdesignates the person HM1 on the tracking screen TRW, i.e., indicates aposition of the tracking screen TRW. The tracking point is associatedwith data on a tracking position and a tracking time (for example, referto FIG. 16B as explained below). The tracking position refers tothree-dimensional coordinates that represent the position in the actualspace corresponding to the position on the tracking screen TRW where theperson HM1 is designated.

Further, the tracking screen TRW refers to a specific screen among ascreen (hereinafter, referred to as camera screen) where the imageobtained by the imaging of the camera apparatus (for example, the cameraapparatus C1) is displayed in the display apparatus 35, which representsa screen on which the person HM1, for example, is projected as amonitoring target that is a target of sound tracking process (to bedescribed later). In the following respective embodiments, a screen onwhich the person HM1 or the like is not projected as the monitoringtarget is referred to as a camera screen, and a screen on which theperson HM1 or the like is projected as the monitoring target is referredto as a tracking screen. The camera screen and the tracking screen aredistinctly used unless otherwise described.

In FIG. 1, for ease of description, it is assumed that the same personHM1 is moving, and thus, z coordinates of the tracking positions attracking points TP1 and TP2 are equal to each other. (See FIG. 1, It isassumed that heights (z axis) of tracking points are the same.) Further,even when the person HM1 moves from the tracking position A1 to thetracking position A2, the person HM1 is imaged by the camera apparatusC1, but the camera apparatus C1 may continuously image the person HM1according to the movement of the person HM1, or may stop the imaging.(See FIG. 1, The camera apparatus may or may not perform imaging whiletracking movement of monitoring target (person HM1).)

An omni-directional microphone array apparatus M1 collects a sound fromthe person HM1, and transmits the sound data that is collected to thedirectivity control apparatus 3 connected through the network NW.

When the person HM1 that is the monitoring target is stationary at thetracking position A1, the directivity control apparatus 3 formsdirectivity of the collected sound in a directivity direction from theomni-directional microphone array apparatus M1 toward the trackingposition A1. Further, when the person HM1 moves from the trackingposition A1 to the tracking position A2, the directivity controlapparatus 3 switches and forms the directivity of the collected soundinto a directivity direction from the omni-directional microphone arrayapparatus M1 toward the tracking position A2. (See FIG. 1, Directivityof collected sound is formed by switching the direction fromomni-directional microphone array apparatus M1 toward tracking positionA1 (three-dimensional position in actual space) of tracking point TP1(designated position on screen) to the direction from omni-directionalmicrophone array apparatus M1 toward tracking position A2 (threedimensional position in actual space) of tracking point TP2 (designatedposition on screen).

In other words, according to the movement of the person HM1 that is themonitoring target from the tracking position A1 to the tracking positionA2, the directivity control apparatus 3 tracking-controls, i.e.,performs a sound tracking process on the directivity of the collectedsound from the direction from the omni-directional microphone arrayapparatus M1 toward the tracking position A1 to the direction from theomni-directional microphone array apparatus M1 toward the trackingposition A2.

The directivity control system 100 shown in FIG. 2 includes one or morecamera apparatuses C1, . . . , Cn, one or more omni-directionalmicrophone array apparatuses M1, . . . , Mm, the directivity controlapparatus 3 and a recorder 4. Here, n and m represent an integer of 1 orgreater, which may be the same, or may be different from each other.This is similarly applied to the following respective embodiments.

The camera apparatus C1, . . . , Cn, the omni-directional microphonearray apparatuses M1, . . . , Mm, the directivity control apparatus 3and the recorder 4 are connected to each other through the network NW.The network NW may be a wired network (for example, an intranet or theInternet), a wireless network (for example, a wireless local areanetwork (LAN), WiMAX (registered trademark) or a wireless wide areanetwork (WAN)). Hereinafter, in the present embodiment, for ease ofdescription, a configuration in which one camera apparatus C1 and oneomni-directional microphone array apparatus M1 are provided is used.

Hereinafter, the respective apparatuses that form the directivitycontrol system 100 will be described. In the respective embodimentsincluding the present embodiment, a housing of the camera apparatus C1and a housing of the omni-directional microphone array apparatus M1 aremounted at different positions separately, but the housing of the cameraapparatus C1 and the housing of the omni-directional microphone arrayapparatus M1 may be integrally mounted at the same position.

The camera apparatus C1 that is an example of an imaging unit is fixedlymounted to a ceiling surface of an event venue, for example, and has afunction as a monitoring camera in a monitoring system. The cameraapparatus C1 captures an image in a predetermined field angle of thecamera apparatus C1 in a predetermined sound collecting area (forexample, a predetermined area in the event venue) by a remote controloperation from a monitoring control chamber (not shown) connected to thenetwork NW. The camera apparatus C1 may be a camera having a PTZfunction, or may be a camera capable of performing imaging inomni-directions. When the camera apparatus C1 is the camera capable ofperforming imaging in omni-directions, the camera C1 transmits imagedata (that is, omni-directional image data) indicating omni-directionalimages in the sound collecting area or planar image data generated byperforming a predetermined distortion correction process for theomni-directional images data to perform panorama conversion to thedirectivity control apparatus 3 or to the recorder 4 through the networkNW.

If an arbitrary position in image data displayed in the displayapparatus 35 is designated by a cursor CSR or a finger FG of the user,the camera apparatus C1 receives coordinate data on the designatedposition in the image data from the directivity control apparatus 3,calculates data on a distance and a direction (including a horizontalangle and a vertical angle, which is the same with the followingdescription) from the camera apparatus C1 to a sound position in anactual space corresponding to the designated position (hereinafter,simply referred to as a “sound position”), and transmits the calculateddata to the directivity control apparatus 3. Since the process ofcalculating the data on the distance and the direction in the cameraapparatus C1 is a known technique in the related art, its description isnot made herein.

The omni-directional microphone array apparatus M1 that is an example ofa sound collecting unit is fixedly mounted to a ceiling surface of anevent venue, for example, and includes at least the following, amicrophone part in which plural microphone units 22 and 23 (see FIGS.36A to 36E) are provided at equal intervals and a central processingunit (CPU) that controls operations of the respective microphone units22 and 23 of the microphone part.

If a power is supplied, the omni-directional microphone array apparatusM1 performs a predetermined sound signal processing (for example, anamplification process, a filtering process and an addition process) forsound data on a sound collected by the microphone element of themicrophone unit, and transmits the sound data obtained by thepredetermined sound signal processing to the directivity controlapparatus 3 or the recorder 4 through the network NW.

Here, the appearance of the housing of the omni-directional microphonearray apparatus M1 will be described with reference to FIGS. 36A to 36E.FIGS. 36A to 36E are appearance drawings of the housing of theomni-directional microphone array apparatus M1. In omni-directionalmicrophone array apparatuses M1C, M1A, M1B, M1 and M1D shown in FIGS.36A to 36E, the appearances and arrangement positions of pluralmicrophone units are different from each other, but the functions of theomni-directional microphone array apparatuses are the same.

The omni-directional microphone array apparatus M1C shown in FIG. 36Ahas a disk shaped housing 21. In the housing 21, the plural microphoneunits 22 and 23 are concentrically arranged. Specifically, the pluralmicrophone units 22 are arranged in a concentric circle shape having thesame center as the housing 21 along the circumference of the housing 21,and the plural microphone units 23 are arranged in a concentric circleshape having the same center as the housing 21 on the inside of thehousing 21. The respective microphone units 22 are arranged with a wideinterval, each of which has a large diameter and has a characteristicsuitable for a low sound range. On the other hand, the respectivemicrophone units 23 are arranged in a narrow interval, each of which hasa small diameter and has a characteristic suitable for a high soundrange.

The omni-directional microphone array apparatus M1A shown in FIG. 36Bhas the disk shaped housing 21. In the housing 21, the plural microphoneunits 22 are arranged in a cross shape along two directions of alongitudinal direction and a transverse direction at equal intervals, inwhich the longitudinal array and the transverse array cross each otherat the center of the housing 21. In the omni-directional microphonearray apparatus M1A, the plural microphone units 22 are linearlyarranged in two directions of the longitudinal direction and thetransverse direction, and thus, it is possible to reduce an arithmeticamount in formation of the directivity of the sound data. In theomni-directional microphone array apparatus M1A shown in FIG. 36B, theplural microphone units 22 may be arranged on only one line in thelongitudinal direction or in the transverse direction.

The omni-directional microphone array apparatus M1B shown in FIG. 36Chas a disk shaped housing 21B having a small diameter, compared with theomni-directional microphone array apparatus M1C shown in FIG. 36A. Inthe housing 21B, the plural microphone units 22 are arranged at equalintervals along the circumference of the housing 21B. Since the intervalof the respective microphone units 22 is short, the omni-directionalmicrophone array apparatus M1B shown in FIG. 36C has a characteristicsuitable for a high sound range.

The omni-directional microphone array apparatus M1 shown in FIG. 36D hasa housing 21C which is in a donut shape or a ring shape and in which anopening portion 21 a having a predetermined diameter is formed at thecenter of the housing 21C. In the directivity control systems 100 and100A according to the present embodiment, the omni-directionalmicrophone array apparatus M1 shown in FIG. 36D is used. In the housing21C, the plural microphone units 22 are concentrically arranged at equalintervals along the circumferential direction of the housing 21C.

The omni-directional microphone array apparatus M1D shown in FIG. 36Ehas a rectangular housing 21D. In the housing 21D, the plural microphoneunits 22 are arranged at equal intervals along the circumference of thehousing 21D. In the omni-directional microphone array apparatus M1Dshown in FIG. 36E, since the housing 21D has the rectangular shape, itis possible to simplify mounting of the omni-directional microphonearray apparatus M1D even at a corner or on a wall surface, for example.

The microphone units 22 and 23 of the omni-directional microphone arrayapparatus M1 may be a non-directional microphone, a bi-directionalmicrophone, a unidirectional microphone, a sharp directional microphone,a super-directional microphone (for example, a shotgun microphone), or acombination thereof.

For example, the directivity control apparatuses 3 and 3A may be astationary PC installed in a monitoring control chamber (not shown), ormay be a data communication terminal that can be carried by a user, suchas a mobile phone, a personal digital assistant (PDA), a tablet terminalor a smart phone.

The directivity control apparatus 3 includes at least the following, acommunication unit 31, an operation unit 32, a memory 33, a signalprocessing unit 34, a display apparatus 35 and a speaker 36. The signalprocessing unit 34 includes at least the following, a directivitydirection calculating unit 34 a, an output control unit 34 b and atracking processing unit 34 c.

The communication unit 31 receives the image data transmitted from thecamera apparatus C1 or the sound data transmitted from theomni-directional microphone array apparatus M1, and outputs the receivedimage data or sound data to the signal processing unit 34.

The operation unit 32 corresponds to a user interface (UI) for notifyingthe signal processing unit 34 of a user's input operation, and theoperation unit 32 is, for example, a pointing device such as a mouse ora keyboard. Further, the operation unit 32 may be disposed correspondingto a display screen of the display apparatus 35, and may be formed usinga touch panel capable of detecting an input operation by means of afinger FG of the user or a stylus pen.

The operation unit 32 outputs coordinate data on the designated positiondesignated by the cursor CSR using a mouse operation of the user or thefinger FG of the user in the image data (that is, the image dataobtained by the imaging of the camera apparatus C1) displayed in thedisplay apparatus 35 to the signal processing unit 34.

The memory 33 is configured by a random access memory (RAM), forexample, and functions as a work memory when the respective units of thedirectivity control apparatus 3 are operated. Further, the memory 33that is an example of an image storage unit or a sound storage unit isconfigured by a hard disk or a flash memory, for example, and stores theimage data or the sound data stored in the recorder 4, that is, theimage data obtained by the imaging of the camera apparatus C1 or thesound data collected by the omni-directional microphone array apparatusM1 for a predetermined period of time.

The memory 33 that is an example of a designation list storage unitstores data on a tracking list LST (for example, see FIG. 16B) that isan example of a designation list that includes data on all thedesignated positions on the tracking screen TRW of the image datadisplayed in the display apparatus 35 and designation times (to bedescribed later).

The signal processing unit 34 is configured by a central processing unit(CPU), a micro processing unit (MPU) or a digital signal processor(DSP), for example, and performs a control process for controllingoverall operations of the respective units of the directivity controlapparatus 3, a data input/output process with respect to the otherunits, a data computation (calculation) process and a data storageprocess.

If the coordinate data on the designated position of the image datadesignated by the cursor CSR using the mouse operation of the user orthe finger FG of the user is obtained from the operation unit 32 duringcalculation of directivity direction coordinates (θ_(MAh), θ_(MAv)), thedirectivity direction calculating unit 34 a transmits the coordinatedata to the camera apparatus C1 through the communication unit 31. Thedirectivity direction calculating unit 34 a obtains data on the distanceand the direction from the mounting position of the camera apparatus 1to the sound (sound source) position in the actual space correspondingto the designated position of the image data, through the communicationunit 31.

The directivity direction calculating unit 34 a calculates thedirectivity direction coordinates (θ_(MAh), θ_(MAv)) from the mountingposition of the omni-directional microphone array apparatus M1 towardthe sound position using the data on the distance and the direction fromthe mounting position of the camera apparatus C1 to the sound position.

Further, as in the present embodiment, when the housing of the cameraapparatus C1 and the housing of the omni-directional microphone arrayapparatus M1 are separately mounted, the directivity directioncalculating unit 34 a calculates the directivity direction coordinates(θ_(MAh), θ_(MAv)) from the omni-directional microphone array apparatusM1 to the sound position (sound source position), using data on apredetermined calibration parameter that is calculated in advance andthe data on the direction (horizontal angle and vertical angle) from thecamera apparatus C1 to the sound position (sound source position). Here,the calibration refers to an operation of calculating or obtaining apredetermined calibration parameter necessary for calculating thedirectivity direction coordinates (θ_(MAh), θ_(MAv)) by the directivitydirection calculating unit 34 a of the directivity control apparatus 3.A specific calibration method and content of the calibration parameterare not particularly limited, and for example, may be realized in arange of a known technique.

Further, when the housing of the omni-directional microphone arrayapparatus M1 is integrally mounted to surround the housing of the cameraapparatus C1, the direction (horizontal angle and vertical angle) fromthe camera apparatus C1 to the sound position (sound source position)may be used as the directivity direction coordinates (θ_(MAh), θ_(MAv))from the omni-directional microphone array apparatus 2 to the soundposition.

Here, among the directivity direction coordinates (θ_(MAh), θ_(MAv)),θ_(MAh) represents the horizontal angle in the directivity directionfrom the mounting position of the omni-directional microphone arrayapparatus 2 toward the sound position, and θ_(MAv) represents thevertical angle in the directivity direction from the mounting positionof the omni-directional microphone array apparatus 2 toward the soundposition. In the following description, for ease of description, it isassumed that reference directions (0 degree direction) of the respectivehorizontal angles of the camera apparatus C1 and the omni-directionalmicrophone array apparatus M1 are identical to each other.

The output control unit 34 b controls the operations of the displayapparatus 35 and the speaker 36. For example, the output control unit 34b that is an example of a display control unit allows the displayapparatus 35 to display the image data transmitted from the cameraapparatus C1 according to an input operation based on the cursor CSRusing the mouse operation of the user or the finger FG of the user, forexample. When the sound data transmitted from the omni-directionalmicrophone array apparatus 2 or the sound data collected by theomni-directional microphone array apparatus M1 for a predeterminedperiod of time is obtained from the recorder 4, the output control unit34 b that is an example of a sound output control unit outputs the sounddata to the speaker 36 according to an input operation based on thecursor CSR using the mouse operation of the user or the finger FG of theuser, for example.

Further, when the image data obtained by the imaging of the cameraapparatus C1 for a predetermined time is obtained from the recorder 4,the output control unit 34 b that is an example of an image reproductionunit allows the display apparatus 35 to reproduce the image dataaccording to the input operation based on the cursor CSR using the mouseoperation of the user or the finger FG of the user, for example.

Further, the output control unit 34 b that is an example of adirectivity forming unit forms the directivity (beam) of the sound(collected sound) collected by the omni-directional microphone arrayapparatus 2 in the directivity direction indicated by the directivitydirection coordinates (θ_(MAh), θ_(MAv)) calculated by the directivitydirection calculating unit 34 a, using the sound data transmitted fromthe omni-directional microphone array apparatus 2 or the sound dataobtained from the recorder 4.

Thus, the directivity control apparatus 3 can relatively increase avolume level of the sound emitted by the monitoring target (for example,the person HM1) that is present in the directivity direction where thedirectivity is formed, and can decrease the sound in a direction wherethe directivity is not formed to relatively reduce the volume level.

The tracking processing unit 34 c that is an example of an informationobtaining unit obtains information on the above-described sound trackingprocess. For example, when a new position is designated according to theinput operation based on the cursor CSR using the mouse operation of theuser or the finger FG of the user, for example, on the tracking screenTRW of the display apparatus 35 in which the image data obtained by theimaging of the camera apparatus C1 is displayed, the tracking processingunit 34 c obtains information on the newly designated position.

Here, the information on the newly designated position includescoordinate information on a time when the new position is designated(designation time) and the sound position (sound source position) wherethe monitoring target (for example, the person HM1) is present in theactual space corresponding to the position on the image data designatedat the designation time, or information on the distance from theomni-directional microphone array apparatus M1 to the sound position(sound source position), in addition to the coordinate informationindicating the position on the image data designated on the trackingscreen TRW.

Further, the tracking processing unit 34 c that is an example of areproduction time calculation unit calculates a sound reproduction timeat a position on a designated movement route according to the inputoperation based on the cursor CSR using the mouse operation of the useror the finger FG of the user, for example, using the data on thetracking list LST stored in the memory 33 (to be described later).

The display apparatus 35 that is an example of a display unit isconfigured by a liquid crystal display (LCD) or an organicelectroluminescence (EL), for example, and displays the image dataobtained by the imaging of the camera apparatus C1 under the control ofthe output control unit 34 b.

The speaker 36 that is an example of a sound output unit outputs sounddata on the sound collected by the omni-directional microphone arrayapparatus M1 or sound data on which the directivity is formed in thedirectivity direction indicated by the directivity direction coordinates(θ_(MAh), θ_(MAv)). The display apparatus 35 and the speaker 36 may havea configuration different from those of the directivity controlapparatus 3.

The recorder 4 stores in association the image data obtained by theimaging of the camera apparatus C1 and the sound data on the soundcollected by the omni-directional microphone array apparatus M1.

The directivity control system 100A shown in FIG. 3 includes one or morecamera apparatuses C1, . . . , Cn, one or more omni-directionalmicrophone array apparatuses M1, . . . , Mm, a directivity controlapparatus 3A, and the recorder 4. In FIG. 3, the same reference numeralsare given to the same components and operations as in the respectiveunits shown in FIG. 2, and thus, the description thereof will besimplified or not repeated and only the contents that are different willbe described.

The directivity control apparatus 3A includes at least the following,the communication unit 31, the operation unit 32, the memory 33, asignal processing unit 34A, the display apparatus 35, the speaker 36,and an image processing unit 37. The signal processing unit 34A includesat least the following, the directivity direction calculating unit 34 a,the output control unit 34 b, the tracking processing unit 34 c, and asound source detecting unit 34 d.

The sound source detecting unit 34 d detects the sound position (soundsource position) in the actual space corresponding to the soundgenerated by the person HM1 that is the monitoring target, from theimage data displayed in the display apparatus 35. For example, the soundsource detecting unit 34 d divides a sound collecting area of theomni-directional microphone array apparatus M1 into plurallattice-shaped areas, and measures the strength of the sound or thesound volume level of the sound on which the directivity is formed fromthe omni-directional microphone array apparatus M1 with respect to thecentral position of each lattice-shaped area. The sound source detectingunit 34 d estimates that the sound source is present in thelattice-shaped area with the highest sound strength or sound volumelevel among all the lattice-shaped areas. The detection result of thesound source detecting unit 34 d includes information on the distancefrom the omni-directional microphone array apparatus M1 to the centralposition of the lattice-shaped area with the highest sound strength orsound volume level.

The image processing unit 37 performs a predetermined image processing(for example, a video motion detection process of detecting a motion ofthe person HM1, a detection process of a person's face or facedirection, or a person detection process) for the image data displayedin the display apparatus 35 according to an instruction of the signalprocessing unit 34, and outputs the image processing result to thesignal processing unit 34.

Further, according to the input operation based on the cursor CSR usingthe mouse operation of the user or the finger FG of the user, forexample, the image processing unit 37 detects an outline DTL of the faceof the monitoring target (for example, the person HM1) displayed in thedisplay apparatus 35, and performs a masking process for the face.Specifically, the image processing unit 37 calculates a rectangularregion that includes the detected outline DTL of the face, and performsa predetermined process of shading off the detected outline DTL in therectangular region (see FIG. 22C). FIG. 22C is a diagram illustrating aprocess of shading off the detected outline DTL of the person's face.The image processing unit 37 outputs the image data generated by theshading-off process to the signal processing unit 34.

FIG. 37 is a diagram briefly illustrating a delay and sum technique inwhich the omni-directional microphone array apparatus M1 forms thedirectivity of the sound data in the direction at an angle θ. For easeof understanding, it is assumed that microphone elements 221 to 22 n arelinearly arranged. In this case, the directivity is present in atwo-dimensional region in the surface. Here, in forming the directivityin a three-dimensional space, the microphones may be two-dimensionallyarranged, and then, the same processing method may be performed.

A sound wave emitted from a sound source 80 is incident at a certainspecified angle (incident angle=(90−θ) degrees) with respect to therespective microphone elements 221, 222, 223, . . . , 22(n−1) and 22 nbuilt in the microphone units 22 and 23 of the omni-directionalmicrophone array apparatus M1.

The sound source 80 is a monitoring target (for example, the person HM1)that is present in the directivity direction of the omni-directionalmicrophone array apparatus M1, for example, and is present in thedirection of a predetermined angle θ with respect to the surface of thehousing 21 of the omni-directional microphone array apparatus M1.Further, an interval d between the respective microphone elements 221,222, 223, . . . , 22(n−1) and 22 n is uniform.

The sound wave emitted from the sound source 80 first reaches themicrophone element 221 to be collected, and then reaches the microphoneelement 222 to be collected. Similarly, the sound wave is sequentiallycollected, and finally reaches the microphone element 22 n to becollected.

A direction from the position of each of the microphone elements 221,222, 223, . . . , 22(n−1) and 22 n of the omni-directional microphonearray apparatus M1 toward the sound source 80 is the same as thedirection from each microphone (microphone element) of theomni-directional microphone array apparatus 2 toward the sound position(sound source position) corresponding to the designated position thatthe user designates in the display apparatus 35 when the sound source 80is the sound emitted from the monitoring target (for example, the personHM1), for example.

Here, from the times when the sound wave sequentially reaches themicrophone elements 221, 222, 223, . . . , and 22(n−1) to the time whenthe sound wave reaches the final microphone element 22 n, arrival timedifferences τ1, τ2, τ3, . . . , and τ(n−1) occur. Thus, when the sounddata on the sound collected by the respective microphone elements 221,222, 223, . . . , 22(n−1) and 22 n is added as is, the addition isperformed in a state where the phase is deviated, and thus, the soundvolume level of the sound wave is weakened as a whole.

Here, τ1 corresponds to a time difference between the time when thesound wave reaches the microphone element 221 and the time when thesound wave reaches the microphone element 22 n, and τ2 corresponds to atime difference between the time when the sound wave reaches themicrophone element 222 and the time when the sound wave reaches themicrophone element 22 n. Similarly, τ1(n−1) corresponds to a timedifference between the time when the sound wave reaches the microphoneelement 22(n−1) and the time when the sound wave reaches the microphoneelement 22 n.

In the present embodiment, the omni-directional microphone arrayapparatus M1 includes A/D converters 241, 242, 243, . . . , 24(n−1) and24 n provided corresponding to the microphone elements 221, 222, 223, .. . , 22(n−1) and 22 n, delay units 251, 252, 253, . . . , 25(n−1) and25 n, and an adder 26 (see FIG. 37).

That is, the omni-directional microphone array apparatus M1 AD-convertsanalogue sound data collected by the microphone elements 221, 222, 223,. . . , 22(n−1) and 22 n into digital sound data in the AD converters241, 242, 243, . . . , 24(n−1), and 24 n.

Further, the omni-directional microphone array apparatus M1 assignsdelay times corresponding to the arrival time differences in therespective microphone elements 221, 222, 223, . . . , 22(n−1) and 22 n,in the delay units 251, 252, 253, . . . , 25(n−1) and 25 n, to align thephases of all sound waves, and then adds sound data after the delayprocess in the adders 26. Thus, the omni-directional microphone arrayapparatus M1 is able to form the directivity of the sound data in thedirection of the predetermined angle θ with respect to the respectivemicrophone elements 221, 222, 223, . . . , 22(n−1) and 22 n.

For example, in FIG. 37, respective delay times D1, D2, D3, . . .,D(n−1) and Dn set in the delay units 251, 252, 253, . . . , 25(n−1) and25 n respectively correspond to the arrival time differences τ1, τ2, τ3,. . . and τ(n−1), which are expressed by the following expression (1).

$\begin{matrix}{\left\lbrack {{Expression}\mspace{14mu} 1} \right\rbrack \mspace{596mu}} & \; \\{{{D\; 1} = {\frac{L\; 1}{Vs} = \frac{\left\{ {{\times \left( {n - 1} \right) \times \cos \mspace{14mu} \theta}} \right\}}{Vs}}}{{D\; 2} = {\frac{L\; 2}{Vs} = \frac{\left\{ {{\times \left( {n - 2} \right) \times \cos \mspace{14mu} \theta}} \right\}}{Vs}}}{{{D\; 3} = {\frac{L\; 3}{Vs} = \frac{\left\{ {{\times \left( {n - 3} \right) \times \cos \mspace{14mu} \theta}} \right\}}{Vs}}},\ldots \mspace{14mu},{{{D\; n} - 1} = {\frac{{L\; n} - 1}{Vs} = \frac{\left\{ {{\times 1 \times \cos \mspace{14mu} \theta}} \right\}}{Vs}}}}{{Dn} = {0\mspace{14mu} \ldots}}} & (1)\end{matrix}$

L1 represents a difference between sound wave arrival distances in themicrophone element 221 and the microphone element 22 n. L2 represents adifference between sound wave arrival distances in the microphoneelement 222 and the microphone element 22 n. L3 represents a differencebetween sound wave arrival distances in the microphone element 223 andthe microphone element 22 n. Similarly, L(n−1) represents a differencebetween sound wave arrival distances in the microphone element 22(n−1)and the microphone element 22 n. “Vs” represents a sound wave velocity(sound velocity). L1, L2, L3, . . . , L(n−1) and Vs are values that arealready known. In FIG. 37, the delay time Dn set in the delay unit 25 nis 0 (zero).

As described above, by changing the delay times D1, D2, D3, . . . , Dn−1and Dn set in the delay units 251, 252, 253, . . . , 25(n−1) and 25 n,the omni-directional microphone array apparatus M1 can easily form thedirectivity of the sound data on the sound collected by the respectivemicrophone elements 221, 222, 223, . . . , 22(n−1) and 22 n built in themicrophone units 22 and 23.

In the above description, it is assumed that the directivity formingprocess shown in FIG. 37 is performed by the omni-directional microphonearray apparatus 2 for ease of description, and this may be similarlyapplied to a different omni-directional microphone array apparatus (forexample, an omni-directional microphone array apparatus Mm). Here, whenthe output control unit 34 b of the signal processing units 34 and 34Aof the directivity control apparatuses 3 and 3A includes the ADconverters 241 to 24 n and the delay units 251 to 25 n of the samenumber as the number of the microphones of the omni-directionalmicrophone array apparatus M1, and one adder 26, the output control unit34 b of the signal processing units 34 and 34A of the directivitycontrol apparatuses 3 and 3A may perform the directivity forming processshown in FIG. 37 using the sound data on the sound collected by eachmicrophone element of the omni-directional microphone array apparatusM1.

(Description of Various Modes and Various Methods)

Here, various modes and various methods common to the respectiveembodiments including the present embodiment will be described indetail.

In the respective embodiments including the present embodiment, thefollowing various modes and various methods are present. The descriptionthereof will be briefly made as follows.

(1) Recording and reproduction mode: On/Off

(2) Tracking mode: On/Off

(3) Tracking processing mode: Manual/Automatic

(4) Tracking target number: Single/Multiple

(5) Manual designation method: Click operation/Drag operation

(6) Slow reproduction mode: On/Off

(7) Enlarge display mode: On/Off

(8) Sound privacy protection mode: On/Off

(9) Image privacy protection mode: On/Off

(10) Connection mode: Each time/Batch

(11) Correction mode: On/Off

(12) Plural-camera switching method: Automatic/Manual

(13) Plural-microphone switching method: Automatic/Manual

(14) Tracking point upper limit setting mode: On/Off

(1) The recording and reproduction mode is used when image data on imageobtained by the imaging of the camera apparatus C1 for a predeterminedperiod of time, for example, is reproduced for content confirmation orthe like by a user (for example, a monitor, which is the same with thefollowing description) at a certain time after being imaged. If therecording and reproduction mode is turned off, the image data on theimage obtained by the imaging of the camera apparatus C1 in real time isdisplayed in the display apparatus 35.

(2) The tracking mode is used when a directivity tracking control (soundtracking process) of the sound collected by the omni-directionalmicrophone array apparatus M1 is performed according to movement of themonitoring target (for example, the person HM1).

(3) The tracking processing method refers to a method for setting theposition (for example, the designated position on the tracking screenTRW of the display apparatus 35 or the position in the actual space)when the directivity tracking control (sound tracking process) of thesound collected by the omni-directional microphone array apparatus M1 isperformed by the movement of the monitoring target (for example, theperson HM1), which is divided into a manual tracking process and anautomatic tracking process. The details thereof will be provided later.

(4) The tracking target number represents the number of monitoringtargets that are targets of the directivity tracking control (soundtracking process) of the sound collected by the omni-directionalmicrophone apparatus M1, which is, for example, singular or in plural ifthe monitoring target is a person.

(5) The manual designation process is used when the user designatestracking points on the tracking screen TRW in the manual trackingprocess (to be described later), which is, for example, a clickoperation or a drag operation of the cursor CSR using the mouseoperation, or a touch operation or a touch slide operation using thefinger FG of the user.

(6) The slow reproduction mode is used when a reproduction speed of theimage data reproduced in the display apparatus 35 is a speed valuesmaller than an initial value (for example, a normal value) on theassumption that the recording and reproduction mode is turned on.

(7) The enlarge display mode is used when the monitoring target (forexample, the person HM1) displayed on the tracking screen TRW of thedisplay apparatus 35 is enlarged and displayed.

(8) The sound privacy protection mode is used, when the sound datacollected by the omni-directional microphone array apparatus M1 isoutput in the speaker 36, or when sound processing (for example, a voicechange process) for making it difficult to recognize a target outputsound is performed.

(9) The image privacy protection mode is used when an image processingfor making it difficult to specify the monitoring target (for example,the person HM1) displayed on the tracking screen TRW of the displayapparatus 35 is performed.

(10) The connection mode is used when designated positions (for example,see a point marker MR1 to be described later) designated on the trackingscreen TRW by manual designation or automatic designation in the courseof movement of the monitoring target are connected. If the connectionmode is performed every time, whenever the designated position isdesignated in the course of movement of the monitoring target, adjacentpoint markers are connected to each other. If the connection mode isperformed in a batch, point markers corresponding to all the designatedpositions obtained in the course of movement of the monitoring targetare connected in a batch between the adjacent point markers.

(11) The correction mode is used when the automatic tracking process isswitched to the manual tracking process in a case where the designatedposition that is automatically designated in the manual tracking processis deviated from the movement course of the monitoring target, or insimilar cases.

(12) The plural-camera switching method is used when the cameraapparatus used for capturing the image of the monitoring target amongthe plural camera apparatuses C1 to Cn is switched. Details of theplural-camera switching method will be described in a second embodiment.

(13) The plural-microphone switching method is used when theomni-directional microphone array apparatus used for the soundcollection emitted by the monitoring target among the pluralomni-directional microphone array apparatuses M1 to Mm is switched.Details of the plural-microphone switching method will be described inthe second embodiment.

(14) The tracking point upper limit setting mode is used when an upperlimit value of the tracking points is set. For example, when thetracking point upper limit setting mode is turned on, if the number oftracking points reaches the upper limit value, the tracking processingunit 34 cmay reset (delete) all the tracking points, or may displayinformation indicating that the number of tracking points reaches theupper limit value on the tracking screen TRW. Further, until the numberof tracking points reaches the upper limit value, it is possible toexecute the sound tracking process plural times.

The designation of the various modes or various methods of (1) to (14)as described above is determined by performing a click operation of thecursor CSR using the mouse operation of the user or a touch operationusing the finger FG of the user, for example, with respect to apredetermined setting button or setting menu in a monitoring systemapplication (not shown), or a setting button or a setting menu displayedon the tracking screen TRW.

Next, an operation example of the manual tracking process in thedirectivity control apparatuses 3 and 3A will be described withreference to FIG. 4. FIG. 4 is a diagram illustrating an operationexample of the manual tracking process.

In FIG. 4, the movement course of the person HM1 that is the monitoringtarget is shown on the tracking screen TRW displayed in the displayapparatus 35, and for example, three tracking points b1, b2 and b3 aredesignated by the click operation or the drag operation of the cursorCSR using the mouse operation of the user, for example.

The tracking processing unit 34 c obtains information on the trackingtime t1 when the cursor CSR designates the tracking point b1, thetracking time t2 when the cursor CSR designates the tracking point b2,and a tracking time t3 when the cursor CSR designates the tracking pointb3. Further, the tracking processing unit 34 c stores coordinateinformation on the tracking point b1 on the tracking screen TRW or threedimensional coordinates that represent a position in the actual spacecorresponding to the coordinate position and information on the trackingtime t1 in association with each other in the memory 33. Further, thetracking processing unit 34 c stores coordinate information on thetracking point b2 on the tracking screen TRW or three dimensionalcoordinates that represent a position in the actual space correspondingto the coordinate position and information on the tracking time t2 inassociation with each other in the memory 33. Further, the trackingprocessing unit 34 c stores coordinate information on the tracking pointb3 on the tracking screen TRW or three dimensional coordinates thatrepresent a position in the actual space corresponding to the coordinateposition and information on the tracking time t3 in association witheach other in the memory 33.

The output control unit 34 b allows the display apparatus 35 to displaya point marker MR1 on the tracking point b1 on the tracking screen

TRW, allows the display apparatus 35 to display a point marker MR2 onthe tracking point b2 on the tracking screen TRW, and allows the displayapparatus 35 to display a point marker MR3 on the tracking point b3 onthe tracking screen TRW. Thus, the output control unit 34 b can clearlydisplay the tracking points through which the moving person HM1 passeson the tracking screen TRW as a track.

Further, the output control unit 34 b connects the point markers MR1 andMR2 to display a moving route LN1, and connects the point markers MR2and MR3 to display a moving route LN2. Next, an operation example of thecorrection mode in the directivity control apparatuses 3 and 3A will bedescribed with reference to FIG. 5. FIG. 5 is a diagram illustrating anoperation example where a tracking point is changed by the manualtracking process when a tracking point automatically designated in theautomatic tracking process is wrong.

On the tracking screen TRW on the left side in FIG. 5, a tracking pointautomatically designated by the image processing unit 37 or the soundsource detecting unit 34 d is different from a point of the person HM1in the course of movement, and a wrong moving route LNW is displayed byconnection of the point markers MR1 and MR2W.

When the correction mode is turned on, as shown in the tracking screenTRW on the right side in FIG. 5, the automatic tracking process isswitched to the manual tracking process. Thus, for example, if a correcttracking point is designated by a click operation using the cursor CSR,the output control unit 34 b connects the point markers MR1 and MR2R todisplay a correct moving route LNR on the tracking screen TRW.

Next, the slow reproduction process in the recording and reproductionmode and the slow reproduction mode in the directivity controlapparatuses 3 and 3A will be described with reference to FIG. 6. FIG. 6is a diagram illustrating the slow reproduction process in the recordingand reproduction mode and the slow reproduction mode.

On the tracking screen TRW on the upper side in FIG. 6, it is assumedthat since the motion of the person HM1 is fast, for example, it isdifficult to designate the person HM1 in either of the manual trackingprocess or the automatic tracking process. When the recording andreproduction mode and the slow reproduction mode are turned on, forexample, if a slow reproduction button displayed in the displayapparatus 35 is touched by the finger FG of the user, the output controlunit 34 b slowly reproduces image data of video indicating the movementcourse of the person HM1 on the tracking screen TRW (see the trackingscreen TRW on the lower side in FIG. 6) with a speed value smaller thanan initial value (normal value) of the reproduction velocity.

Thus, since the output control unit 34 b can delay the motion of theperson HM1 on the tracking screen TRW, it is possible to easilydesignate the tracking point in the manual tracking process or theautomatic tracking process. When the movement speed of the person HM1 isequal to or greater than a predetermined value, the output control unit34 b may perform the slow reproduction process without accepting thetouch operation of the finger FG of the user. Further, the reproductionspeed during the slow reproduction may be constant, or may beappropriately changed according to the input operation based on thecursor CSR using the mouse operation of the user or the finger FG of theuser.

Next, the enlarge display process in the enlarge display mode in thedirectivity control apparatuses 3 and 3A will be described withreference to FIG. 7. FIG. 7 is a diagram illustrating the enlargedisplay process in the enlarge display mode.

On the tracking screen TRW on the upper side in FIG. 7, it is assumedthat since the size of the person HM1 is small, for example, it isdifficult to designate the person HM1 in the manual tracking process orthe automatic tracking process. For example, after the enlarge displaymode is turned on by the click operation of the cursor CSR using themouse operation of the user, for example, if the click operation isperformed on a position (display position) of the person HM1, the outputcontrol unit 34 b enlarges and displays the tracking screen TRW at apredetermined magnification while placing the clicked position (see thetracking screen TRW on the lower side in FIG. 7) at a center ofenlargement. Thus, since the output control unit 34 b can enlarge anddisplay the person HM1 on the tracking screen TRW, it is possible toeasily designate the tracking point in the manual tracking process orthe automatic tracking process.

Here, the output control unit 34 b may enlarge and display the contentof the tracking screen TRW on a separate popup screen (not shown) whileplacing the clicked position at a center of enlargement. Thus, theoutput control unit 34 b can compare the tracking screen TRW that is notenlarged and displayed with the popup screen that is enlarged anddisplayed by a simple designation operation of the user, so that theuser can easily designate the monitoring target (person HM1).

Further, when the tracking point is not yet designated, for example, theoutput control unit 34 b may enlarge and display the content of theprojected camera screen with reference to the center of the displayapparatus 35. Thus, for example, when the monitoring target (the personHM1) is projected around the center of the display apparatus 35, theoutput control unit 34 b can allow the user to easily designate themonitoring target by a simple designation operation of the user, forexample.

Further, when plural monitoring targets are designated, the outputcontrol unit 34 b may enlarge and display the monitoring targets whileplacing a position corresponding to a geometrical average of the pluraldesignated positions at a center on the tracking screen TRW. Thus, theoutput control unit 34 b can allow the user to easily select the pluralmonitoring targets projected on the tracking screen TRW.

Next, the automatic scroll process after the enlarge display process inthe enlarge display mode in the directivity control apparatuses 3 and 3Awill be described with reference to FIGS. 8A, 8B and 8C. FIG. 8A is adiagram illustrating the automatic scroll process after the enlargedisplay process in the enlarge display mode. FIG. 8B is a diagramillustrating a tracking screen TRW at the time t=t1. FIG. 8C is adiagram illustrating a tracking screen TRW at the time t=t2.

In FIG. 8A, in an imaging area C1RN of the camera apparatus C1, amovement path from the position at the time t=t1 of the person HM1 thatis the monitoring target to the position at the time t=t2 is shown. Forexample, as the tracking screen TRW is enlarged and displayed,consequently, an overall image of the imaging area C1RN may not beprojected onto the tracking screen TRW.

The output control unit 34 b automatically scrolls the tracking screenTRW so that the person HM1 is constantly displayed at the center of thetracking screen TRW, along the movement path of the person HM1 from thetime t=t1 to the time t=t2, for example, according to the inputoperation based on the cursor CSR of the mouse operation of the user orthe finger of the user. Thus, as the person HM1 projected to theenlarged and displayed tracking screen TRW is moving, the output controlunit 34 b automatically scrolls the tracking screen TRW so that thedesignated position of the user is constantly present at the center ofthe tracking screen TRW. Thus, it is possible to prevent the designatedposition of the person HM1 of the user from being deviated from thetracking screen TRW even though the tracking screen TRW is enlarged anddisplayed, and thus, it is possible to easily designate the person HM1during continuous movement on the tracking screen TRW. (See FIGS. 8A-8C,When the tracking point is manually designated after enlarge display,automatic scroll is performed so that tracking point (for example,position around person HM1) is not deviated from the display area and ispresent at the center of the display area.)

In FIG. 8B, the tracking screen TRW at the time t=t1 is shown, in whichthe person HM1 is displayed at the center thereof. TP1 in FIG. 8Brepresents a tracking point where the person HM1 is designated by aninput operation based on the cursor CSR using the mouse operation of theuser or the finger FG of the user at time t=t1.

Similarly, in FIG. 8C, the tracking screen TRW at the time t=t2 isshown, in which the person HM1 is displayed at the center thereof. TP2in FIG. 8C represents a tracking point where the person HM1 isdesignated by an input operation based on the cursor CSR using the mouseoperation of the user or the finger FG of the user at time t=t2. In bothof FIGS. 8B and 8C, since the person HM1 that is the monitoring targetis displayed at the center on the tracking screen TRW during theautomatic scroll process, the selection of the user becomes easy.

Next, an overall flow of the manual tracking process in the directivitycontrol system 100 according to the present embodiment will be describedwith reference to FIGS. 9A and 9B. FIG. 9A is a flowchart illustrating afirst example of the overall flow of the manual tracking process in thedirectivity control system 100 according to the first embodiment. FIG.9B is a flowchart illustrating a second example of the overall flow ofthe manual tracking process in the directivity control system 100according to the first embodiment.

Hereinafter, for ease of description, first, the overall flow of themanual tracking process in the directivity control system 100 accordingto the present embodiment will be described with reference to FIGS. 9Aand 9B, and then, the detailed description for the contents ofindividual processes with reference to the drawings will be made later.In the operation shown in FIG. 9B, the same step numbers are given tothe same content as in the operation shown in FIG. 9A, and thedescription thereof will be simplified or not repeated and only thecontents that are different will be described. In FIGS. 9A and 9B, theoperation of the directivity control apparatus 3 is shown.

For description of FIG. 9A, it is assumed that the output control unit34 b forms the directivity of the collected sound in a direction fromthe omni-directional microphone array apparatus M1 toward the position(sound position or sound source position) of the person HM1corresponding to the position designated by the input operation based onthe cursor CRS using the mouse operation of the user or the finger FG ofthe user, on the tracking screen TRW of the display apparatus 35 wherethe image of the person HM1 that is the monitoring target obtained bythe imaging of the camera apparatus C1 is projected. This is similarlyassumed for description of FIG. 9B.

In FIG. 9A, if the tracking mode is turned off (S1, NO), the manualtracking process shown in FIG. 9A is terminated. However, if thetracking mode is turned on (S1, YES), the tracking auxiliary process isstarted (S2). The details about the tracking auxiliary process will bedescribed with reference to FIG. 13A.

After step S2, on the tracking screen TRW of the display apparatus 35,the tracking position of the movement course (movement path) of theperson HM1, that is, the tracking point is designated by the clickoperation of the cursor CSR using the mouse operation of the user or thetouch operation of the finger FG of the user (S3).

The tracking processing unit 34 c stores three dimensional coordinatesthat represent a position in the actual space corresponding to thedesignated position on the tracking screen TRW and the designation timedesignated in step S3 in the memory 33 in association as the trackingposition and the tracking time of each tracking point, and displays thepoint marker on the tracking point on the tracking screen TRW throughthe output control unit 34 b (S4). The point marker may be displayed bythe tracking processing unit 34 c, which is similarly applied to thefollowing respective embodiments.

The output control unit 34 b forms the directivity of the collectedsound in a direction from the omni-directional microphone arrayapparatus M1 toward the position (sound position, sound source position)of the person HM1 corresponding to the tracking point designated in stepS3 (S5). Here, when the tracking processing unit 34 c has only to obtainthe data on the tracking position and the tracking time of the trackingpoint according to the input operation based on the cursor CSR using themouse operation of the user or the finger FG of the user, the operationin step S5 may not be performed. In other words, the output control unit34 b may not switch the directivity in a direction from theomni-directional microphone array apparatus M1 toward the position(sound position, sound source position) of the person HM1 correspondingto the tracking point designated in step S3, which is similarly appliedto the following respective embodiments.

After step S5, the output control unit 34 b performs the trackingconnection process (S6). Details of the tracking connection process willbe described later with reference to FIG. 15A. After step S6, the outputcontrol unit 34 b outputs the collected sound in which the directivityis formed in step S5 through the speaker 36 (S7). Details of the soundoutput processing will be described later with reference to FIG. 21A.After step S7, the operation of the directivity control apparatus 3returns to step S1, and the processes of steps S1 to S7 are repeateduntil the tracking mode is turned off.

In FIG. 9B, after step S1, the tracking auxiliary process is started(S2). Details of the tracking auxiliary process will be described laterwith reference to FIG. 13A. After step S2, on the tracking screen TRW ofthe display apparatus 35, with respect to the tracking position of themovement course (movement path) of the person HM1, that is, the trackingpoint, the drag operation of the cursor CSR using the mouse operation ofthe user or the touch slide operation of the finger FG of the user isstarted (S3A)

After step S3A, if a predetermined time (for example, about severalseconds) does not elapse after storage of data on the tracking positionand the tracking time corresponding to the previous tracking point isfinished (S8, NO), it is considered that the drag operation or the touchslide operation started in step S3A is not terminated, and the operationof the directivity control apparatus 3 proceeds to step S7.

On the other hand, after step S3, if the predetermined time (forexample, about several seconds) elapses after storage of data on thetracking position and the tracking time corresponding to the previoustracking point is finished (S8, YES), it is considered that the dragoperation or the touch slide operation started in step S3 is terminated,and a new tracking point is designated. That is, the tracking processingunit 34 c stores three dimensional coordinates that represent theposition in the actual space corresponding to the designated positionand the designation time when the drag operation or the touch slideoperation is terminated in the memory 33 in association as a trackingposition and a tracking time of the new tracking point, and displays thepoint marker on the tracking point on the tracking screen TRW throughthe output control unit 34 b (S4). Since the operations after step S4 isthe same as the operations after step S4 shown in FIG. 9A, thedescription thereof will not be repeated.

Next, an overall flow of the automatic tracking process in thedirectivity control system 100A according to the present embodiment willbe described with reference to FIGS. 10A and 10B, FIGS. 11A and 11B andFIG. 12. FIG. 10A is a flowchart illustrating a first example of anoverall flow of the automatic tracking process in the directivitycontrol system 100A according to the first embodiment. FIG. 10B is aflowchart illustrating the first example of the automatic trackingprocess shown in FIG. 10A. FIG. 11A is a flowchart illustrating a secondexample of the automatic tracking process shown in FIG. 10A. FIG. 11B isa flowchart illustrating an example of a tracking correction processshown in FIG. 11A. FIG. 12 is a flowchart illustrating a third exampleof the automatic tracking process shown in FIG. 10A.

Further, in FIG. 10A, similarly to FIGS. 9A and 9B, for ease ofdescription, first, the overall flow of the automatic tracking processin the directivity control system 100A according to the presentembodiment will be described with reference to FIG. 10A, and then,detailed contents of individual processes will be described withreference to the following drawings.

In the operation shown in FIG. 10A, the same step numbers are given tothe same content as in the operation shown in FIG. 9A or 9B, and thedescription thereof will be simplified or not repeated and only thecontents that are different will be described. In FIG. 10A, theoperation of the directivity control apparatus 3 is shown.

In the description of FIG. 10A, it is assumed that the output controlunit 34 b forms the directivity of the collected sound in a directionfrom the omni-directional microphone array apparatus M1 toward theposition (sound position or sound source position) of the person HM1corresponding to the position automatically designated using thedetection processing result of the sound source detecting unit 34 d orthe image processing unit 37, on the tracking screen TRW of the displayapparatus 35 on which the image of the person HM1 who is the monitoringtarget obtained by the imaging of the camera apparatus C1 is projected.

In FIG. 10A, after step S1, the tracking auxiliary process is started

(23). Details of the tracking auxiliary process will be described laterwith reference to FIG. 13A. After step S3B, the automatic trackingprocess is performed (S3B). Details of the automatic tracking processwill be described later with reference to FIGS. 10A, 11A and 12. Afterstep S3B, the output control unit 34 b forms the directivity of thecollected sound in a direction from the omni-directional microphonearray apparatus M1 toward the position (sound position, sound sourceposition) of the person HM1 corresponding to the tracking pointautomatically designated in step S3B (S5). Further, since the operationin step S5 is the same as the operation after step S4 shown in FIG. 9A,the description thereof will not be repeated.

In FIG. 10B, the image processing unit 37 determines whether the personHM1 that is the monitoring target is detected on the tracking screen TRWof the display apparatus 35 by performing a known image processing, andif it is determined that the person HM1 is detected, the imageprocessing unit 37 outputs the determination result (including data onthe detected position of the person HM1 (for example, a knownrepresentative point) and the detection time) to the tracking processingunit 34 c of the signal processing unit 34 (S3B-1).

Alternatively, the sound source detecting unit 34 d determines whetherthe position of the sound (sound source) emitted by the person HM1 thatis the monitoring target is detected on the tracking screen TRW of thedisplay apparatus 35 by performing a known sound source detectionprocess, and if it is determined that the position of the sound sourceis detected, the sound source detecting unit 34 d outputs thedetermination result (including data on the detected position of thesound source and the detection time) to the tracking processing unit 34c (S3B-1). For ease of description of step S3B-1, it is assumed that amonitoring target other than the person HM1 that is the monitoringtarget is not present on the tracking screen TRW.

The tracking processing unit 34 c automatically sets the designatedposition of the person HM1 in the automatic tracking process, that is,the tracking point using the determination result of the imageprocessing unit 37 or the sound source detecting unit 34 d (S3B-1). Thetracking processing unit 34 c stores three dimensional coordinates thatrepresent the position in the actual space corresponding to thedetection position and the detection time on the tracking screen TRWautomatically designated in step S3B-1 in the memory 33 in associationas a tracking position and a tracking time of each tracking point, anddisplays the point marker on the tracking point on the tracking screenTRW through the output control unit 34 b (S3B-2). After step S3B-2, theautomatic tracking process shown in FIG. 10B is terminated, and then,the procedure proceeds to step S5 shown in FIG. 10A.

In FIG. 11A, if an initial tracking point (initial position) is alreadydesignated (S3B-3, YES), the operation of step S3B-4 is not performed.On the other hand, if the initial tracking point is not designated(S3B-3, NO), the position of the movement course (movement path) of theperson HM1, that is, the tracking point is designated on the trackingscreen TRW of the display apparatus 35 by the input operation (forexample, click operation, touch operation) based on the cursor CSR usingthe mouse operation of the user or the finger FG of the user (S3B-4).

If the initial tracking point is already designated, after the initialtracking point is designated in step S3B-4, the tracking processing unit34 c automatically designates the next tracking point using thedetermination result of the image processing unit 37 or the sound sourcedetecting unit 34 d while placing the initial tracking point at a center(S3B-5). Thus, for example, since the user designates the initialtracking point, the detection process of the information on the positionof the sound (sound source) emitted by the person HM1 or the informationon the position of the person HM1 is started while placing the initialtracking point (initial position) at a center on the tracking screenTRW, and thus, the tracking processing unit 34 c is able to perform therespective detection processes at high speed.

The tracking processing unit 34 c stores in association threedimensional coordinates that represent the position in the actual spacecorresponding to the detected position and the detection time on thetracking screen TRW that are automatically designated in step S3B-5 inmemory 33 as the tracking position and the tracking time of the trackingpoint, and displays the point marker on the tracking point on thetracking screen TRW through the output control unit 34 b (S3B-2).

After step S3B-2, if the correction operation of the tracking point isnot performed (53B-6, NO), the automatic tracking process shown in FIG.11A is terminated, and then, the procedure proceeds to step S3 shown inFIG. 10A.

On the other hand, after step S3B-2, for example, if the operation ofcorrecting the tracking position corresponding to the tracking point isperformed since the determination result of the image processing unit 37or the sound source detecting unit 34 d is wrong (S3B-6, YES), thetracking correction process shown in FIG. 11B is performed (S3B-7).

In FIG. 11B, when the sound emitted by the person HM1 who is moving onthe tracking screen TRW is output, the output of the sound istemporarily stopped by the input operation based on the cursor CSR usingthe mouse operation of the user or the finger FG of the user (S3B-7-1).After step S3B-7-1, the process is temporarily changed from theautomatic tracking process to the manual tracking process as thecorrection mode is turned on by the input operation based on the cursorCSR using the mouse operation of the user or the finger FG of the user,and a correct tracking point is designated (S3B-7-2).

The output control unit 34 b deletes the wrong point marker displayed onthe tracking screen TRW designated in step S3B-7-2 (S3B-7-3), displaysthe point marker on the changed tracking point, that is, the trackingpoint designated in step S3B-7-2, and restarts the output of the soundthat is temporarily stopped in step S3B-7-1 (S3B-7-3). Further, thetracking processing unit 34 c saves the position designated in stepS3B-7-2 as the tracking point (S3B-7-3). After step S3B-7-3, thetracking correction process shown in FIG. 11B is terminated, and then,the procedure proceeds to step S5 shown in FIG. 10A.

In FIG. 12, the image processing unit 37 determines whether the personHM1 that is the monitoring target is detected on the tracking screen TRWof the display apparatus 35 by performing known image processing(S3B-8). If it is determined that the person HM1 is detected (S3B-9,YES), the image processing unit 37 calculates the detected position (forexample, a known representative point) of the person HM1, and outputseach data on the detection time and the detected position to thetracking processing unit 34 c of the signal processing unit 34 as thedetermination result (S3B-10).

The sound source detecting unit 34 d determines whether the position ofthe sound (sound source) emitted by the person HM1 that is themonitoring target is detected on the tracking screen TRW of the displayapparatus 35 by performing a known sound detection process, and if it isdetermined that the position of the sound source is detected, the soundsource detecting unit 34 d calculates the detected position of theperson HM1, and outputs each data on the detection time and the detectedposition to the tracking processing unit 34 c as the determinationresult (S3B-11).

The tracking processing unit 34 c stores the detected position and thedetection time of the sound source on the tracking screen TRW calculatedin the step S3B-11 in the memory 33 in association as the trackingposition and the tracking time of the tracking point, and displays thepoint marker around the tracking position on the tracking screen TRWthrough the output control unit 34 b (S3B-12).

After step S3B-12, the tracking processing unit 34 c determines whetherthe distance between the detected position of the person HM1 calculatedin step S3B-10 and the detected position of the sound source calculatedin step S3B-11 is within a predetermined value (S3B-13). If it isdetermined that the distance between the detected position of the personHM1 and the detected position of the sound source is within thepredetermined value (S3B-13, YES), the automatic tracking process shownin FIG. 12 is terminated, and then, the procedure proceeds to step S5shown in FIG. 10A.

On the other hand, if it is determined that the distance between thedetected position of the person HM1 and the detected position of thesound source is not within the predetermined value (S3B-13, NO), theautomatic tracking process shown in FIG. 11B is performed (S3B-7). Sincethe tracking correction process is described with reference to FIG. 11B,the description thereof will not be made herein. After step S3B-7, theautomatic tracking process shown in FIG. 12 is terminated, and then, theprocedure proceeds to step S5 shown in FIG. 10A.

Thus, if the distance between the position of the sound source and theposition of the person HM1 detected by the detection process of theposition of the sound source or the detection process of the position ofthe person HM1 is equal to or greater than the predetermined value, forexample, the tracking processing unit 34 c can easily modify and obtainthe information on the position designated by the change operation ofthe position of the user in the tracking correction process (see FIG.11B) as the information on the position of the person HM1. Further, ifthe distance between the position of the sound source and the positionof the person HM1 detected by the detection process of the position ofthe sound source or the detection process of the position of the personHM1 is not equal to or greater than the predetermined value, forexample, the tracking processing unit 34 c can easily obtain theposition of the sound source or the position of the person HM1 as theinformation on the position after movement of the person HM1, withoutperforming the change operation of the position of the user, forexample.

Next, details of the tracking auxiliary process in the directivitycontrol apparatuses 3 and 3A will be described with reference to FIG.13A. FIG. 13A is a flowchart illustrating an example of the trackingauxiliary process shown in FIG. 9A.

In FIG. 13A, if the enlarge display mode of the directivity controlapparatuses 3 and 3A is turned off (S2-1, NO), the operation of thedirectivity control apparatuses 3 and 3A proceeds to step S2-5. On theother hand, when the enlarge display mode of the directivity controlapparatuses 3 and 3A is turned on (S2-1, YES), the directivity controlapparatuses 3 and 3A perform the image privacy protection process(S2-2), and perform the automatic scroll process (S2-3). Details of theimage privacy protection process will be described with reference toFIG. 21B. Details of the automatic scroll process will be described withreference to FIGS. 13B, 14A and 14B.

After step S2-3, the output control unit 34 b enlarges and displays thecontent of the tracking screen TRW at a predetermined magnificationwhile placing a tracking position corresponding to an immediately closetracking point at a center on the tracking screen TRW (S2-4). After stepS2-4, if both of the recording and reproduction mode and the slowreproduction mode of the directivity control apparatuses 3 and 3A areturned on (S2-5, YES), the output control unit 34 b slowly reproducesimage data on video indicating the movement course of the person HM1with a speed value smaller than the initial value (normal value) of thereproduction speed on the tracking screen TRW (S2-6).

After step S2-6 or if both of the recording and reproduction mode andthe slow reproduction mode of the directivity control apparatuses 3 and3A are not turned on (S2-5, NO), the tracking auxiliary process shown inFIG. 13A is terminated, and then, the procedure proceeds to step S5shown in FIG. 9A, step S3A shown in FIG. 9B or step S3B shown in FIG.10A.

Next, details of the automatic scroll process in the directivity controlapparatuses 3 and 3A will be described with reference to FIG. 13B, 14Aand 14B. FIG. 13B is a flowchart illustrating an example of theautomatic scroll process shown in FIG. 13A. FIG. 14A is a flowchartillustrating an example of an automatic scroll process necessitydetermination process shown in FIG. 13B. FIG. 14B is a diagramillustrating a scroll necessity determination line in the automaticscroll process necessity determination process.

In FIG. 13B, the tracking processing unit 34 c performs the automaticscroll process necessity determination process (S2-3-1). Details of theautomatic scroll process necessity determination process will bedescribed later with reference to FIG. 14A.

After step S2-3-1, if it is determined that it is necessary to performthe automatic scroll process as the automatic scroll process necessitydetermination process result (S2-3-2, YES), the output control unit 34 bperforms a predetermined automatic scroll process for the trackingscreen TRW (S2-3-3). For example, the output control unit 34 bautomatically scrolls the tracking screen TRW so that the person HM1 isconstantly displayed at the center of the tracking screen TRW along themovement path of the person HM1 on the tracking screen TRW, according tothe input operation based on the CSR using the mouse operation of theuser or the finger FG of the user. Thus, even when the tracking screenTRW is enlarged and displayed, the output control unit 34 b can preventthe designated position of the person HM1 that is the monitoring targetof the user from being deviated from the tracking screen TRW, and caneasily designate the person HM1 during continuous movement on thetracking screen TRW.

When the tracking point is not designated at step S2-3-1-1, the trackingscreen TRW may be automatically scrolled so that the person HM1 isdisplayed at the center of the tracking screen TRW, and in this case,the output control unit 34 b may not perform the automatic scrollprocess necessity determination process shown in step S2-3-1.

Further, when the person HM1 moves out of a scroll determination lineJDL (to be described later), the output control unit 34 b performs theautomatic scroll process by a predetermined amount in the movementdirection of the person HM1 (for example, in a direction out of wherethe person HM1 is out of the scroll determination line JDL (to bedescribed later)). Thus, even when the tracking screen TRW is enlargedand displayed, the output control unit 34 b can prevent the designatedposition of the person HM1 that is the monitoring target of the userfrom being deviated from the tracking screen TRW.

Further, when the person HM1 moves out of the scroll determination lineJDL (to be described later), the output control unit 34 b performs theautomatic scroll process so that the position (for example, a nexttracking point) designated by the input operation based on the cursorCSR using the mouse operation of the user or the finger FG of the useris present at the center of the tracking screen TRW. Thus, even when thetracking screen TRW is enlarged and displayed, the output control unit34 b can prevent the designated position of the person HM1 that is themonitoring target of the user from being deviated from the trackingscreen TRW, and can easily designate the person HM1 during continuousmovement on the tracking screen TRW.

After step S2-3-3, or if it is determined that the automatic scrollprocess is not necessary as the automatic scroll process necessitydetermination process result (S2-3-2, NO), the automatic scroll processshown in FIG. 13B is terminated, and then, the procedure proceeds tostep S2-4 shown in FIG. 13A.

In FIG. 14A, the tracking processing unit 34 c determines whether thetracking position corresponding to the tracking point TP1 that isdesignated is out of any one of the upper, lower, right and leftdetermination lines JDL on the tracking screen XTRW that is enlarged anddisplayed (S2-3-1-1).

If it is determined that the tracking position is not out of any scrolldetermination line JDL (S2-3-1-1, NO), the tracking processing unit 34 cdetermines that the automatic scroll process is not necessary(S2-3-1-2). On the other hand, if it is determined that the trackingposition is out of any scroll determination line JDL (S2-3-1-1, YES),the tracking processing unit 34 c determines that the automatic scrollprocess is necessary, and stores the type (for example, informationindicating any one of four scroll determination lines JDL shown in FIG.14B) of the scroll determination line JDL in the memory 33 (S2-3-1-3).After steps S2-3-1-2 and S2-3-1-3, the automatic scroll processnecessity determination process shown in FIG. 14A is terminated, andthen, the procedure proceeds to step S2-3-2 shown in FIG. 13B.

Next, details of the tracking connection process in the directivitycontrol apparatuses 3 and 3A will be described with reference to FIGS.15A and 15B. FIG. 15A is a flowchart illustrating an example of thetracking connection process shown in FIG. 9A. FIG. 15B is a flowchartillustrating an example of a batch connection process shown in FIG. 15A.

In FIG. 15A, if the tracking point is already designated (S6-1, YES),the tracking processing unit 34 c determines whether the connection modeis performed every time (S6-2). If it is determined that the connectionmode is performed every time (S6-2, YES), the output control unit 34 bconnects one or more tracking points that are designated immediatelybefore with the latest one or more tracking points for display (S6-3).Thus, since at least the current designated position and the lastdesignated position among the plural designated positions designated bythe designation operation of the user when the person HM1 projected ontothe tracking screen TRW of the display apparatus 35 is moving areconnected to each other for display, the output control unit 34 b canclearly show the track of a part of the movement of the person HM1.

In step S6-3, the present invention is not limited to the operation ofthe single designation where the tracking points are designated one byone, and also includes an operation where the plural tracking points aresimultaneously designated. This is similarly applied to step S6-4-3 tobe described later.

After step S6-3, or if it is determined that the tracking point is notyet designated (S6-1, NO), the tracking connection process shown in FIG.15A is terminated, and then, the procedure proceeds to step S7 shown inFIGS. 9A and 9B or FIG. 10A.

If it is determined that the connection mode is not performed every time(S6-2, NO), the batch connection process is performed (S6-4). The batchconnection process will be described with reference to FIG. 15B.

In FIG. 15B, the tracking processing unit 34 c sequentially reads dataon the tracking list LST (for example, see FIG. 16B) stored in thememory 33 (S6-4-1, YES). If it is determined that the read data is astart point of the tracking point (S6-4-2, YES), the tracking processingunit 34 c reads again the data on the tracking list LST (for example,see FIG. 16B) (S6-4-1).

On the other hand, if it is determined that the read data is not thestart point of the tracking point (S6-4-2, NO), the output control unit34 b connects point markers of the last designated one or more trackingpoints and the latest one or more tracking points using the read data onthe tracking list for display (S6-4-3).

After step S6-4-3, if the connection is performed up to an end point ofthe tracking point (S6-4-4, YES), the batch connection process shown inFIG. 15B is terminated, and then, the procedure proceeds to step S7shown in FIGS. 9A and 9B or FIG. 10A.

On the other hand, after step S6-4-3, if the connection is not performedup to an end point of the tracking point (S6-4-4, NO), the trackingprocessing unit 34 c sequentially reads the data on the tracking listLST (for example, see FIG. 16B) stored in the memory 33, and then, theoperations from step S6-4-1 to step S6-4-4 are repeated until the pointmarkers corresponding to all the tracking points in the tracking listLST are connected to each other for display. Thus, since the outputcontrol unit 34 b connects one or two designated positions adjacent toeach designated position for display with respect to all the pluraldesignated positions designated by the designation operation of the userwhen the person HM1 projected onto the tracking screen TRW of thedisplay apparatus 35 is moving, it is possible to clearly display thetrack of the entire movement of the person HM1.

FIG. 16A is a diagram illustrating a reproduction start time PT of acollected sound corresponding to a designated position P0 of a user on amovement route between tracking points displayed with respect toone-time movement of the person HM1. FIG. 16B is a diagram illustratinga first example of a tracking list. In FIG. 16A, TP1, TP2, TP3 and TP4represent tracking points designated during one-time movement of theperson HM1 as shown in the tracking list LST shown in FIG. 16B.

In FIG. 16B, coordinates (x, y, z) indicating the tracking position andthe tracking time are stored in association for each of the trackingpoints TP1 (start point), TP2, TP3 and TP4 (end point). For ease ofdescription, it is assumed that a z coordinate value z0 of the zcoordinate indicating the tracking position is constant. (see FIGS. 16Aand 16B, It is assumed that height direction (x-axis) is constant.)

If the designated position P0 is designated according to the inputoperation based on the cursor CSR using the mouse operation of the useror the finger FG of the user on the movement route between the trackingpoints shown in FIG. 16A, the tracking processing unit 34 c extracts twotracking points TP1 and TP2 before and after the designated position P0,and calculates the reproduction start time PT at the designated positionP0 according to expression (2) using the data on the coordinatesindicating the tracking positions of the tracking points TP1 and TP2 andthe tracking times. (See FIGS. 16A and 16B, Two tracking points TP1 andTP2 are extracted immediately after designated position P0, andreproduction time PT is calculated at designated position P0 usinginformation on positions and times of two extracted tracking points TP1and TP2.)

$\begin{matrix}\left\lbrack {{Expression}\mspace{14mu} 2} \right\rbrack & \; \\{{PT} = {{T\; 1} + \left\{ {\left( {{T\; 2} - {T\; 1}} \right) \times \frac{\sqrt{\left( {{x\; 0} - {x\; 1}} \right)^{2} + \left( {{y\; 0} - {y\; 1}} \right)^{2}}}{\sqrt{\left( {{x\; 2} - {x\; 1}} \right)^{2} + \left( {{y\; 2} - {y\; 1}} \right)^{2}}}} \right\}}} & (2)\end{matrix}$

Further, when the sound is output (reproduced) to the speaker 36, theoutput control unit 34 b forms the directivity in the directivitydirection corresponding to the tracking position in the order of thetracking times including the designated position P0 designated by theinput operation based on the cursor CSR using the mouse operation or thefinger FG of the user, and then, outputs (reproduces) the sound to whichthe directivity is formed.

FIG. 17A is a diagram illustrating a reproduction start time PT of acollected sound corresponding to a designated position P0 of a user on amovement route between different tracking points based on pluralsimultaneous designations. FIG. 17B is a diagram illustrating a secondexample of the tracking list LST. In FIG. 17A, (TP11, TP21), (TP12,TP22), (TP13, TP23), and (TP14, TP24) are tracking points that aresimultaneously designated during movement of different persons that areplural monitoring targets, for example, as shown in the tracking listLST shown in FIG. 17B.

In FIG. 17B, for each set of (TP11, TP21), (TP12, TP22), (TP13, TP23),and (TP14, TP24), the coordinates (x, y, z) indicating the trackingposition and the tracking time are stored in association. The trackingpoints (TP11, TP21) are start points, and the tracking points (TP14,TP24) are end points. For ease of description, it is assumed that the zcoordinate value z0 of the z coordinate indicating the tracking positionis constant.

If the designated position P0 is designated according to the inputoperation based on the cursor CSR using the mouse operation of the useror the finger FG of the user at any position on different movementroutes between the tracking points shown in FIG. 17A, the trackingprocessing unit 34 c extracts two tracking points TP11 and TP12 beforeand after the designated position P0, and calculates the reproductionstart time PT at the designated position P0 according to expression (3)using the data on the coordinates indicating the tracking positions ofthe tracking points TP11 and TP12 and the tracking times.

$\begin{matrix}\left\lbrack {{Expression}\mspace{14mu} 3} \right\rbrack & \; \\{{PT} = {{T\; 1} + \left\{ {\left( {{T\; 2} - {T\; 1}} \right) \times \frac{\sqrt{\left( {{x\; 0} - {x\; 11}} \right)^{2} + \left( {{y\; 0} - {y\; 11}} \right)^{2}}}{\sqrt{\left( {{x\; 12} - {x\; 11}} \right)^{2} + \left( {{y\; 1\; 2} - {y\; 11}} \right)^{2}}}} \right\}}} & (3)\end{matrix}$

Further, when the sound is output (reproduced) to the speaker 36, theoutput control unit 34 b forms the directivity in the directivitydirection corresponding to the tracking position in the order of thetracking times including the designated position P0 designated by theinput operation based on the cursor CSR using the mouse operation or thefinger FG of the user, and then, outputs (reproduces) the sound to whichthe directivity is formed.

FIG. 18A is a diagram illustrating reproduction start times PT and PT′of a collected sound corresponding to respective designated positions P0and P0′ of a user on movement routes between different tracking pointsbased on plural-time designations. FIG. 18B is a diagram illustrating athird example of the tracking list LST. In FIG. 18A, (TP11, TP12, TP13and TP14) represent tracking points designated during movement of aperson that is a first monitoring target, for example, as shown in thetracking list LST shown in FIG. 18B. Further, in FIG. 18A, similarly,(TP21, TP22 and TP23) represent tracking points designated duringmovement of a person that is a second monitoring target, for example.The person that is the second monitoring target may be the same personas the first monitoring target, or may be a different person.

In FIG. 18B, for each of TP11, TP12, TP13, TP14, TP21, TP22 and TP23,the coordinates (x, y, z) indicating the tracking position and thetracking time are stored in association. The tracking points TP11 andTP21 are start points, and the tracking points TP14 and TP23 are endpoints. For ease of description, it is assumed that the z coordinatevalue z0 of the z coordinate indicating the tracking position isconstant.

If the designated positions P0 and P0′ are designated according to theinput operation based on the cursor CSR using the mouse operation of theuser or the finger FG of the user at any position on the respectivemovement routes between the tracking points shown in FIG. 18A, thetracking processing unit 34 c extracts two tracking points (TP11, TP12)and two tracking points (TP21, TP22) before and after the designatedpositions P0 and P0′, and calculates the reproduction start times PT andPT′ at the designated positions P0 and P0′ according to expressions (4)and (5) using the data on the coordinates indicating the trackingpositions of the (TP11, TP12) and (TP21, TP22) and the tracking times,respectively. In expressions (4) and (5), the coordinates of thedesignated position P0 is (x0, y0, z0), and the coordinates of thedesignated position P0′ is (x0′, y0′, z0).

$\begin{matrix}\left\lbrack {{Expression}\mspace{14mu} 4} \right\rbrack & \; \\{{PT} = {{T\; 11} + \left\{ {\left( {{T\; 12} - {T\; 1\; 1}} \right) \times \frac{\sqrt{\left( {{x\; 0} - {x\; 11}} \right)^{2} + \left( {{y\; 0} - {y\; 11}} \right)^{2}}}{\sqrt{\left( {{x\; 12} - {x\; 11}} \right)^{2} + \left( {{y\; 12} - {y\; 11}} \right)^{2}}}} \right\}}} & (4) \\\left\lbrack {{Expression}\mspace{14mu} 5} \right\rbrack & \; \\{{PT}^{\prime} = {{T\; 21} + \left\{ {\left( {{T\; 22} - {T\; 2\; 1}} \right) \times \frac{\sqrt{\left( {{x\; 0^{\prime}} - {x\; 21}} \right)^{2} + \left( {{y\; 0^{\prime}} - {y\; 21}} \right)^{2}}}{\sqrt{\left( {{x\; 22} - {x\; 21}} \right)^{2} + \left( {{y\; 22} - {y\; 21}} \right)^{2}}}} \right\}}} & (5)\end{matrix}$

In FIG. 18A, the number of tracking points and the tracking timesdesignated during movement of the first person and the number oftracking points and the tracking times designated during movement of thesecond person may not be identical to each other. (See FIGS. 18A and18B, The number of tracking points and tracking times between the firstand the second may not be identical to each other.) Further, when thesound is output (reproduced) to the speaker 36, the output control unit34 b forms the directivity in the directivity direction corresponding tothe tracking position in the order of the tracking times including thedesignated position P0 or the designated position P0′ designated by theinput operation based on the cursor CSR using the mouse operation or thefinger FG of the user, and then, outputs (reproduces) the sound to whichthe directivity is formed. (See FIGS. 18A and 18B, In sound output(reproduction), data is sorted according to the time, and thedirectivity is formed in the position direction at specific time tooutput sound.)

Next, an overall flow of the movement route display reproduction processin the directivity control apparatuses 3 and 3A where the recording andreduction mode is mainly turned on will be described with reference toFIG. 19A. FIG. 19A is a flowchart illustrating an example of the overallflow of the movement route display reproduction process using thetracking list LST in the directivity control systems 100 and 100Aaccording to the first embodiment.

In FIG. 19A, first, the movement route display process is performed(S11). Details of the movement route display process will be describedlater with reference to FIG. 20. After step S11, if the designatedposition P0 is designated according to the input operation based on thecursor CSR using the mouse operation of the user or the finger FG of theuser on the movement route between the tracking points displayed in stepS11 (S12), the reproduction start time calculation process is performed(S13). Details of the reproduction start time calculation process willbe described later with reference to FIG. 19B.

The tracking processing unit 34 c reads the coordinates of all thetracking positions (or one tracking position) corresponding to thetracking times that are closest to the reproduction start time PT of thedesignated position P0 calculated in the reproduction start timecalculation process shown in step S13, with reference to the trackinglist LST stored in the memory 33 (S14). Further, the output control unit34 b forms the directivity of the collected sound in a direction fromthe omni-directional microphone array apparatus M1 toward the all thetracking positions (or one tracking position) using the data on thecoordinates of the tracking positions read by the tracking processingunit 34 c (S14). Thus, the output control unit 34 b is able to form inadvance, according to a position (arbitrarily designated position) thatis arbitrarily designated by the user on the movement route indicatingthe track of the movement of the person HM1, the directivity of thesound in a direction toward the tracking position designated next timewith reference to the arbitrarily designated position.

After step s14, the output control unit 34 b starts the reproduction ofthe sound data that is collected stored in the recorder 4 or the memory33 from the reproduction start time PT calculated in step S13 (S15).

After step S15, if the next tracking time is present in a predeterminedtime from the reproduction start time PT (S16, YES), the output controlunit 34 b forms the directivity of the collected sound in a directionfrom the omni-directional microphone array apparatus M1 toward all thetracking positions (or one tracking position), using the data on thecoordinates of all the tracking positions (or one tracking position)corresponding to the next tracking time (S17).

After step S17, or if the next tracking time is not present in thepredetermined time from the reproduction start time PT (S16, NO), thesound output process is performed (S7). Details of the sound outputprocess will be described later with reference to FIG. 21A. After stepS7, if the sound output process at the tracking time corresponding tothe end point of the tracking points is terminated (S18, YES), themovement route display reproduction process shown in FIG. 19A isterminated. Thus, the output control unit 34 b can clearly output thecollected sound emitted by the monitoring target at the reproductionstart time calculated according to the position arbitrarily designatedby the user, and can form in advance the directivity of the sound at thenext designated position when the next designated position is present inthe predetermined time from the reproduction start time.

On the other hand, after step S7, if the sound output process at thetracking time corresponding to the end point of the tracking points isnot terminated (S18, NO), the operations from step S16 to step S18 arerepeated until the sound output process at the tracking timecorresponding to the end point of the tracking points is terminated.

Next, details of the reproduction start time calculation process in thedirectivity control apparatuses 3 and 3A will be described withreference to FIG. 19B. FIG. 19B is a flowchart illustrating an exampleof the reproduction start time calculation process shown in FIG. 19A.

In FIG. 19B, the tracking processing unit 34 c reads the tracking listLST (for example, see FIG. 16B) stored in the memory 33 (S13-1). Thetracking processing unit 34 c extracts two tracking points TP1 and TP2before and after the designated position P0 designated in step S12, fromthe data on the tracking list LST read in step S13-1 (S13-2). Thetracking processing unit 34 c calculates the reproduction start time PTat the designated position P0 using the data on the coordinatesindicating the tracking positions of the tracking points TP1 and TP2 andthe tracking times (S13-3, for example, see expression (2)). After stepS13-3, the reproduction start time calculation process shown in FIG. 19Bis terminated, and then, the procedure proceeds to step S14 shown inFIG. 19A.

Next, details of the movement route display process in the directivitycontrol apparatuses 3 and 3A will be described with reference to FIG.20. FIG. 20 is a flowchart illustrating an example of a movement routedisplay process shown in FIG. 19A.

In FIG. 20, the tracking processing unit 34 c sequentially reads thedata on the tracking list LST (for example, see FIG. 16B) stored in thememory 33 (S11-1). If the connection of the point markers for all thetracking points read in step S11-1 is terminated (S11-2, YES), themovement route display process shown in FIG. 20 is terminated, and then,the procedure proceeds to step S12 shown in FIG. 19A.

On the other hand, if the connection of the point markers for all thetracking points read in step S11-1 is not terminated (S11-2, NO), thetracking processing unit 34 c sequentially reads the data on thetracking list LST (for example, see FIG. 16B). The output control unit34 b distinctly displays the point markers for each monitoring target atone or more tracking points read by the tracking processing unit 34 c(S11-3).

In step S11-3, although not particularly shown, for example, accordingto the input operation (for example, a right click operation and a leftclick operation of the mouse, simultaneous depression of plural keys ofa keyboard, click operation and simultaneous depression of numeric keysof the keyboard, simultaneous designation for touch panel or the like)based on the cursor CSR using the mouse operation of the user or thefinger FG of the user, the output control unit 34 b distinctly displaysthe point markers for each monitoring target in a manner capable ofidentifying the same monitoring target (for example, the same sign, anidentification number, a combination of a sign and an identificationnumber, a frame of a predetermined shape, or the like). Here, the frameof the predetermined shape represents a rectangle, a circle or atriangle, for example. For identification, instead of the frame shape, aframe line type (for example, a solid line or a dotted line), a framecolor, a number written on a frame or the like may be used.

After step S11-3, if it is determined that the data on the trackingpoint read in step S11-3 is the start point of the tracking points(S11-4, YES), the tracking processing unit 34 c reads the data on thetracking list LST again (for example, see FIG. 16B) (S11-3).

On the other hand, if it is determined that the data on the trackingpoint read in step S11-3 is not the start point of the tracking points(S11-4, NO), the output control unit 34 b connects the respective pointmarkers of one or more tracking points that are designated immediatelybefore with the latest one or more tracking points using the data on theread tracking list, for display (S11-5).

After step S11-5, if the connection is performed up to the end point ofthe tracking points on the tracking list LST read in step S11-1 (S11-6,YES), the procedure proceeds to the operation of step S11-2.

On the other hand, after step S11-5, if the connection is not performedup to the end point of the tracking points on the tracking list LST readin step S11-1 (S11-6, NO), the operations from step S11-3 to step S11-6are repeated until the connection is performed up to the end point ofthe tracking points on the tracking list LST read in step S11-1.

Next, the sound output process and the image privacy protection processin the directivity control apparatuses 3 and 3A will be described withreference to FIGS. 21A and 21B and FIGS. 22A to 22C, respectively. FIG.21A is a flowchart illustrating an example of the sound output processshown in FIG. 9A. FIG. 21B is a flowchart illustrating an example of theimage privacy protection process shown in FIG. 13A. FIG. 22A is adiagram illustrating an example of a waveform of a sound signalcorresponding to a pitch before a voice change process. FIG. 22B is adiagram illustrating an example of a waveform of a sound signalcorresponding to a pitch after the voice change process. FIG. 22C is adiagram illustrating a shading-off process of an outline of the face ofa detected person.

In FIG. 21A, the output control unit 34 b determines whether the soundprivacy protection mode is turned on (S7-1). If it is determined thatthe sound privacy protection mode is turned on (S7-1, YES), the outputcontrol unit 34 b performs voice change process for the sound data thatis collected output in the speaker 36 (S7-2).

After step S7-2, or if it is determined that the sound privacyprotection mode is turned off (S7-1, NO), the output control unit 34 boutputs the collected sound as it is through the speaker 36 (S7-3).After step S7-3, the sound output process shown in FIG. 21A isterminated, and then, the procedure returns to step S1 shown in FIGS. 9Aand 9B or FIG. 10A. As an example of the voice change process, forexample, the output control unit 34 b increases or decreases a pitch ofa waveform of the sound data on the sound collected by theomni-directional microphone array apparatus M1 or the sound data towhich the output control unit 34 b forms the directivity (for example,see FIGS. 22A and 22B). Thus, since the output control unit 34 bperforms the voice change process for the sound collected in real timeby the omni-directional microphone array apparatus M1 by the simpleinput operation of the user to output the sound, for example, it isdifficult to recognize the target output of the sound emitted by theperson HM1, and thus, it is possible to effectively protect the soundprivacy of the person HM1 that is currently imaged. Further, when thesound collected by the omni-directional microphone array apparatus M1for a predetermined period of time is output according to the user'ssimple input operation, for example, since the output control unit 34 bperforms the voice change process for the sound to output the sound, itis difficult to recognize the target output of the sound emitted by theperson HM1, and thus, it is possible to effectively protect the soundprivacy of the person HM1.

In FIG. 21B, the tracking processing unit 34 c determines whether theimage privacy protection mode is turned on (S-2-1). If it is determinedthat the image privacy protection mode is turned on (S2-2-1, YES), theimage processing unit 37 detects (extracts) the outline DTL of the faceof the person HM1 displayed on the tracking screen TRW of the displayapparatus 35 (S2-2-2), and performs a masking process for the outlineDTL of the face (S2-2-3). Specifically, the image processing unit 37calculates the rectangular region including the detected outline DTL ofthe face, and performs the predetermined shading-off process in therectangular region (see FIG. 22C). The image processing unit 37 outputsthe image data generated by the shading-off process to the outputcontrol unit 34 b.

After step S2-2-3, or if it is determined that the image privacyprotection mode is turned off (S2-2-1, NO), the output control unit 34 ballows the display apparatus 35 to display the image data obtained fromthe image processing unit 37 (S2-2-4).

Thus, since the image processing unit 37 performs the masking processfor a part (for example, the face) of the person HM1 that is themonitoring target projected onto the tracking screen TRW of the displayapparatus 35 according to the user's simple input operation, forexample, it is difficult to recognize the person HM1 that is themonitoring target, and thus, it is possible to effectively protect theprivacy.

As long as the image privacy protection mode of the directivity controlapparatuses 3 and 3A is turned on at the time when the monitoring target(for example, the person HM1) appears on the camera screen, the imageprivacy protection process shown in FIG. 21B may be performed even whenthe enlarge display mode is not turned on.

As described above, in the directivity control system 100 and 100Aaccording to the present embodiment, the directivity control apparatuses3 and 3A form the directivity of the sound in the direction from theomni-directional microphone array apparatus M1 including the pluralmicrophones toward the monitoring target (for example, the person HM1)corresponding to the designated position with respect to the image dataon the tracking screen TRW of the display apparatus 35, and obtains theinformation (for example, the tracking position and the tracking timecorresponding to the tracking point) relating to the designated positionwhere the moving monitoring target (for example, the person HM1) isdesignated. Further, the directivity control apparatuses 3 and 3Aswitch, in a tracking manner, the sound directivity in the directiontoward the monitoring target (for example, the person HM1) correspondingto the designated position, using the information on the designatedposition with respect to the image data on the tracking screen TRW ofthe display apparatus 35.

Thus, even when the monitoring target (for example, the person HM1)projected to the image data on the tracking screen TRW of the displayapparatus 35 is moving, since the directivity control apparatuses 3 and3A form the directivity of the sound formed in the direction toward theposition before the movement of the monitoring target (for example, theperson HM1) toward the position after the movement of the monitoringtarget (for example, the person HM1), it is possible to appropriatelyform the directivity of the sound in a tracking manner according to themovement of the monitoring target (for example, the person HM1), andthus, it is possible to prevent deterioration of monitoring workefficiency of the monitor.

Further, the directivity control apparatuses 3 and 3A can easily obtaincorrect information on the position after the movement of the monitoringtarget (for example, the person HM1) according to the simple manualoperation of designating the monitoring target (for example, the personHM1) that is moving in the image data projected onto the tracking screenTRW of the display apparatus 35.

Further, the directivity control apparatus 3A can simply detect thesound source of the sound emitted by the monitoring target (for example,the person HM1), and the monitoring target (for example, the personHM1), from the image data projected onto the tracking screen TRW of thedisplay apparatus 35, and thus, can easily obtain the information on theposition of the sound source and the information on the position of themonitoring target as the information on the position of the monitoringtarget (for example, the person HM1) after movement.

Second Embodiment

In the second embodiment, when the monitoring target (for example, aperson) is out of the imaging area of the camera apparatus or the soundcollecting area of the omni-directional microphone array apparatusaccording to its movement state, a directivity control apparatus 3Bswitches the camera apparatus used for imaging the monitoring target toa different camera apparatus, or switches the omni-directionalmicrophone array apparatus used for the sound collection emitted by themonitoring target to a different omni-directional microphone arrayapparatus.

In the present embodiment, it is assumed that the camera apparatus usedfor capturing the image of the monitoring target (for example, theperson HM1) that is the target of the sound tracking process and theomni-directional microphone array apparatus used for collection of thesound emitted from the person HM1 are associated with each other inadvance and information about the association is stored in the memory 33of the directivity control apparatus 3B in advance.

FIG. 23 is a block diagram illustrating a system configuration exampleof a directivity control system 100B according to a second embodiment.The directivity control system 100B shown in FIG. 23 includes one ormore camera apparatuses C1, . . . , Cn, one or more omni-directionalmicrophone array apparatuses M1, . . . , Mm, the directivity controlapparatus 3B, and the recorder 4. In the description of respective unitsshown in FIG. 23, the same reference numerals are given to the samecomponents and operations as in the respective units of the directivitycontrol systems 100 and 100A shown in FIGS. 2 and 3, and the descriptionthereof will be simplified or not repeated and only the contents thatare different will be described.

The directivity control apparatus 3B may be a stationary PC provided ina monitoring control chamber (not shown), or may be a data communicationterminal that can be carried by a user, such as a mobile phone, apersonal digital assistant (PDA), a tablet terminal or a smart phone.

The directivity control apparatus 3B includes at least the following, acommunication unit 31, an operation unit 32, a memory 33, a signalprocessing unit 34, a display apparatus 35, a speaker 36, an imageprocessing unit 37 and an operation switching control unit 38. Thesignal processing unit 34 includes at least the following, a directivitydirection calculating unit 34 a, an output control unit 34 b, a trackingprocessing unit 34 c, and a sound source detecting unit 34 d.

The operation switching control unit 38 performs various operations forswitching a camera apparatus used for capturing an image of a monitoringtarget of the directivity control system 100B or an omni-directionalmicrophone array apparatus used for collection of a sound emitted by themonitoring target among the plural camera apparatuses C1 to Cn or theplural omni-directional microphone array apparatuses M1 to Mm, based ona variety of information or data relating to a movement situation of themonitoring target (for example, a person) obtained by the trackingprocessing unit 34 c.

Next, an automatic switching process of a camera apparatus in thedirectivity control apparatus 3B will be described with reference toFIG. 24. FIG. 24 is a diagram illustrating an automatic switchingprocess of a camera apparatus used for capturing an image displayed inthe display apparatus 35. In FIG. 24, for ease of description, anexample in which as a person HM1 that is a monitoring target is movingfrom a tracking position A1 to a tracking position A2, the cameraapparatus used for capturing the image of the person HM1 is switchedfrom a camera apparatus C1 to a camera apparatus C2 will be described.

The tracking position A1 is within the range of an imaging area C1RN ofthe camera apparatus C1, which is within the range of a predeterminedswitching determination line JC1 of the camera apparatus C1. Thetracking position A2 is within the range of an imaging area C2RN of thecamera apparatus C2, which is out of the range of the switchingdetermination line JC1 of the camera apparatus C1. Although not shown,the tracking positions A1 and A2 are within the sound collecting area ofthe omni-directional microphone array apparatus M1.

When the person HM1 is out of the imaging area C1RN of the cameraapparatus C1, the operation switching control unit 38 notifies thecamera apparatus C2 of information indicating that the camera apparatusused for capturing the image of the person HM1 is to be switched fromthe camera apparatus C1 to the camera apparatus C2 through thecommunication unit 31 and the network NW. In other words, the operationswitching control unit 38 instructs the camera apparatus C2 to preparefor the capturing of the image in a field angle range of the cameraapparatus C2. At this time, image data on video obtained by the imagingof the camera apparatus C1 is displayed on the tracking screen TRW ofthe display apparatus 35.

For example, when the person HM1 is out of the switching determinationline JC1 of the camera apparatus C1, the operation switching controlunit 38 notifies the camera apparatus C2 of the information indicatingthat the camera apparatus used for capturing the image of the person HM1is to be switched from the camera apparatus C1 to the camera apparatusC2, through the communication unit 31 and the network NW.

The operation switching control unit 38 determines whether the personHM1 is out of the switching determination line JC1 using information onthe distance between the camera apparatus C1 and the person HM1 measuredby the camera apparatus C1. Specifically, when the person HM1 is presentin the field angle of the camera apparatus C1 and the distance from thecamera apparatus C1 to the person HM1 becomes larger than the (known)distance from the camera apparatus C1 to the switching determinationline JC1, the operation switching control unit 38 determines that theperson HM1 is out of the switching determination line JC1. It is assumedthat the operation switching control unit 38 recognizes in advance acamera apparatus (for example, the camera apparatus C2) capable of beingswitched from the camera apparatus C1 and also recognizes in advance acamera apparatus capable of being switched from a different cameraapparatus.

If it is determined that the person HM1 who is out of the switchingdetermination line JC1 is out of the imaging area C1RN of the cameraapparatus C1, the operation switching control unit 38 switches thecamera apparatus used for capturing the image of the person HM1 from thecamera apparatus C1 to the camera apparatus C2. Then, on the trackingscreen TRW of the display apparatus 35, image data (for example, imagedata on the moving person HM1) on the video obtained by the imaging ofthe camera apparatus C2 is displayed.

Thus, the operation switching control unit 38 can adaptively switch thecamera apparatus to a camera apparatus capable of correctly projectingthe image of the moving monitoring target (for example, the person HM1),and can easily designate the image of the monitoring target (forexample, the person HM1) of the user.

Next, the automatic switching process of the omni-directional microphonearray apparatus in the directivity control apparatus 3B will bedescribed with reference to FIG. 25. FIG. 25 is a diagram illustratingan automatic switching process of an omni-directional microphone arrayapparatus used for collection of a sound of a monitoring target (forexample, the person HM1). In FIG. 25, for ease of description, anexample in which as the person HM1 that is the monitoring target ismoving from the tracking position A1 to the tracking position A2, theomni-directional microphone array apparatus used for the soundcollection emitted by the person HM1 is switched from anomni-directional microphone array apparatus M1 to an omni-directionalmicrophone array apparatus M2 will be described.

The tracking position A1 is within the range of a sound collecting areaM1RN of the omni-directional microphone array apparatus M1, which iswithin the range of a switching determination line JM1 of thepredetermined omni-directional microphone array apparatus M1. Thetracking position A2 is within the range of a sound collecting area M2RNof the omni-directional microphone array apparatus M2, which is out ofthe range of the switching determination line JM1 of theomni-directional microphone array apparatus M1. Although not shown, thetracking positions A1 and A2 are within the imaging area of the cameraapparatus C1.

When the person HM1 is out of the sound collecting area M1RN of theomni-directional microphone array apparatus M1, the operation switchingcontrol unit 38 notifies the omni-directional microphone array apparatusM2 of information indicating that the omni-directional microphone arrayapparatus used for the sound collection emitted by the person HM1 is tobe switched from the omni-directional microphone array apparatus M1 tothe omni-directional microphone array apparatus M2 through thecommunication unit 31 and the network NW. In other words, the operationswitching control unit 38 instructs the omni-directional microphonearray apparatus M2 to prepare for the sound collection in the soundcollecting area of the omni-directional microphone array apparatus M2.

For example, when the person HM1 is out of the switching determinationline JM1 of the omni-directional microphone array apparatus M1, theoperation switching control unit 38 notifies the omni-directionalmicrophone array apparatus M2 of the information indicating that theomni-directional microphone array apparatus used for the soundcollection emitted by the person HM1 is to be switched from theomni-directional microphone array apparatus M1 to the omni-directionalmicrophone array apparatus M2, through the communication unit 31 and thenetwork NW.

The operation switching control unit 38 determines whether the personHM1 is out of the switching determination line JM1 using information onthe distance between the omni-directional microphone array apparatus M1and the person HM1. Specifically, when the distance from theomni-directional microphone array apparatus M1 to the person HM1 becomeslarger than the (known) distance from the omni-directional microphonearray apparatus M1 to the switching determination line JM1, theoperation switching control unit 38 determines that the person HM1 isout of the switching determination line JM1. It is assumed that theoperation switching control unit 38 recognizes an omni-directionalmicrophone array apparatus (for example, the omni-directional microphonearray apparatus M2) capable of being switched from the omni-directionalmicrophone array apparatus M1 in advance and also recognizes anomni-directional microphone array apparatus capable of being switchedfrom a different omni-directional microphone array apparatus in advance.

If it is determined that the person HM1 who is out of the switchingdetermination line JM1 is out of the sound collecting area M1RN of theomni-directional microphone array apparatus M1, the operation switchingcontrol unit 38 switches the omni-directional microphone array apparatusM used for the sound collection emitted by the person HM1 from theomni-directional microphone array apparatus M1 to the omni-directionalmicrophone array apparatus M2.

Thus, the operation switching control unit 38 can adaptively switch theomni-directional microphone array apparatus to an omni-directionalmicrophone array apparatus capable of correctly collecting the soundemitted by the moving monitoring target (for example, the person HM1),and can collect the sound emitted by the monitoring target (for example,the person HM1) with high accuracy.

Next, a manual switching process of the camera apparatus in thedirectivity control apparatus 3B will be described with reference toFIG. 26. FIG. 26 is a diagram illustrating the manual switching processof the camera apparatus used for capturing the image displayed in thedisplay apparatus 35. In FIG. 26, in the display apparatus 35, accordingto an input operation based on the cursor CSR using the mouse operationof the user or the finger FG of the user, the tracking screen TRW of theimage obtained by the imaging of the camera apparatus C1 that iscurrently used for capturing the image of the person HM1 is switched toa multi-camera screen that includes a camera screen C1W of the cameraapparatus C1 and camera screens of camera apparatuses (for example,eight camera apparatuses) around the camera apparatus C1. (See FIG. 26,Tracking screen is switched to multi-camera screen including respectivecamera screens of currently used camera apparatus and selectableadjacent camera apparatuses.)

Similarly to FIG. 24, switchable camera apparatuses are determined inadvance for the currently used camera apparatus C1, which are cameraapparatuses C2, C3 and C4, for example. On the multi-camera screen shownin FIG. 26, camera screens C2W, C3W and C4W obtained by the imaging ofthe camera apparatuses C2, C3 and C4 are displayed (see hatched portionsshown in FIG. 26) in the display apparatus 35. It is assumed that theperson HM1 moves in a movement direction MV1.

It is assumed that a user touches any camera screen (for example, thecamera screen C3W) among three camera screens C2W, C3W and C4W by thefinger FG with respect to the multi-camera screen shown in FIG. 26, inconsideration of the movement direction MV1 of the person HM1 that isthe monitoring target.

The operation switching control unit 38 switches the camera apparatusused for capturing the image of the person HM1 from the currently usedcamera apparatus C1 to the camera apparatus C3 corresponding to thecamera screen C3W that is a target of the touch operation, according tothe touch operation of the finger FG of the user. (See FIG. 26, Ifcamera screen is selected by the user operation according to movementdirection of person, the camera apparatus is switched to cameraapparatus corresponding to selected camera screen.)

Thus, the operation switching control unit 38 can adaptively switch thecamera apparatus to a camera apparatus capable of correctly projectingthe image of the moving monitoring target (for example, the person HM1),and can simply designate the image of the monitoring target (forexample, the person HM1) of the user by the user's simple operation.

Next, a manual switching process of the omni-directional microphonearray apparatus in the directivity control apparatus 3B will bedescribed with reference to FIG. 27. FIG. 27 is a diagram illustratingthe manual switching process of the omni-directional microphone arrayapparatus used for the sound collection of the monitoring target (forexample, the person HM1). In FIG. 27, the person HM1 that is themonitoring target is displayed at the center on the tracking screen TRW.Further, it is assumed that omni-directional microphone arrayapparatuses capable of being switched from the currently usedomni-directional microphone array apparatus M1 are threeomni-directional microphone array apparatuses M2, M3 and M4 providedaround the omni-directional microphone array apparatus M1.

In FIG. 27, according to an input operation based on the cursor CSRusing the mouse operation of the user or the finger FG of the user,markers M2R, M3R and M4R indicating rough positions of theomni-directional microphone array apparatuses M2, M3 and M4 capable ofbeing switched from the currently used microphone array apparatus M1 aredisplayed on the tracking screen TRW (see FIG. 27, (1) Markersindicating outline positions of switchable adjacent omnidirectionalmicrophone array apparatuses are displayed on the tracking screen).

The user selects any marker (for example, the marker M3R) among threemarkers by the touch operation of the finger FG of the user, inconsideration of the movement direction MV1 of the person HM1 that isthe monitoring target from the tracking position A1 corresponding to thetracking point on the person HM1 (see FIG. 27, (2) Any one marker isselected by a user operation). The operation switching control unit 38instructs the omni-directional microphone array apparatus M3corresponding to the marker M3R selected by the touch operation of thefinger FG of the user, instead of the currently used omni-directionalmicrophone array apparatus M1, to start the sound collection through thecommunication unit 31 and the network NW (see FIG. 27, (3) Soundcollection is started in omni-directional microphone array apparatuscorresponding to selected marker).

The output control unit 34 b switches a directivity in a direction fromthe omni-directional microphone array apparatus M3 corresponding to theselected marker M3R toward the tracking position of the person HM1 atthe current time (see FIG. 27, (4) Directivity is switched in adirection from omni-directional microphone array apparatus correspondingto the selected marker toward tracking position at the current point).Then, the markers M2R, M3R and M4R indicating the rough positions of theomni-directional microphone array apparatuses M2, M3 and M4 displayed onthe tracking screen TRW are deleted by the output control unit 34 b (seeFIG. 27, (5) Markers indicating outline positions of adjacentomni-directional microphone array apparatuses displayed on trackingscreen are deleted).

Thus, the operation switching control unit 38 can adaptively switch theomni-directional microphone array apparatus to the omni-directionalmicrophone array apparatus M3 capable of correctly collecting the soundemitted by the moving monitoring target (for example, the person HM1),and can collect the sound emitted by the person HM1 according to themovement direction MV1 of the person HM1 with high accuracy by theuser's simple operation on the markers M2R, M3R and M4R displayed on thetracking screen TRW.

Next, a selection process of an optimal omni-directional microphonearray apparatus in the directivity control apparatus 3B will bedescribed with reference to FIG. 28. FIG. 28 is a diagram illustratingthe selection process of the omni-directional microphone array apparatusoptimal for the sound collection of the monitoring target. In thedisplay apparatus 35 on an upper left side in FIG. 28, camera screens ofall the camera apparatuses (for example, nine camera apparatuses)controlled by the directivity control system 100B are displayed in listaccording to an input operation based on the cursor CSR using the mouseoperation of the user or the finger FG of the user.

Among the respective camera screens that are displayed in list in thedisplay apparatus 35 on an upper left side in FIG. 28, camera screens onwhich the monitoring target (for example, the person HM1) that is thetarget of the sound tracking process is projected are the camera screensC1W, C2W and C3W (Upper left side in FIG. 28: Camera screens of allcamera apparatuses are display in the list). It is assumed that thecamera screen C1W where the projection of the person HM1 is the mostexcellent is selected from among the camera screens C1W, C2W and C3W,according to an input operation based on the cursor CSR using the mouseoperation of the user or the finger FG of the user (Upper left side inFIG. 28: Camera screen having the most excellent projection of theperson is selected).

The operation switching control unit 38 selects the camera apparatus C5corresponding to the camera screen C1W as the camera apparatus used forcapturing the image of the person HM1 according to the selection of thecamera screen C1W of the user, for switching. Thus, the output controlunit 34 b enlarges the image data obtained by the imaging of the cameraapparatus corresponding to the camera screen C1W and displays the resulton the tracking screen TRW1 of the display apparatus 35 (see a lowerleft side in FIG. 28: Selected camera screen C1W is enlarged anddisplayed on tracking screen TRW1→Camera apparatus C1 is selected).Further, the output control unit 34 b displays the markers M1R, M2R, M3Rand M4R roughly indicating positions of all the omni-directionalmicrophone array apparatuses associated with the camera apparatus C1selected by the operation switching control unit 38 at four corners ofthe tracking screen TRW1. The display positions of the markers M1R, M2R,M3R and M4R are not limited to the four corners on the tracking screenTRW1. (See upper right side in FIG. 28: Markers indicating outlinepositions of all omni-directional microphone array apparatuses M1, M2,M3 and M4 associated with selected camera apparatus C1 are displayed.)

Further, if the markers M1R, M2R, M3R and M4R are sequentiallydesignated according to the input operation based on the cursor CSRusing the mouse operation of the user or the finger FG of the user, theoutput control unit 34 b forms the directivity in a direction from theomni-directional microphone array apparatus corresponding to each markertoward the position of the person HM1, for each marker, whilehighlighting (for example, by a blink Br) the markers one by one, andoutputs the sound collected for a predetermined time. (See upper rightside in FIG. 28: If Markers are sequentially designated, each collectedsound to which directivity is formed is output for predetermined timewhile highlighting (for example, by blink) marker.)

If the marker (for example, the marker M3R) indicating the roughposition of the omni-directional microphone array apparatus that isdetermined as optimal by the user in the sound output for apredetermined time is selected, the operation switching control unit 38selects the omni-directional microphone array apparatus M3 correspondingto the selected marker M3R as the omni-directional microphone arrayapparatus used for the sound collection emitted by the person HM1, forswitching. (See lower right side in FIG. 28: Marker determined asoptimal by user in sound collected for the predetermined time isselected→Omni-directional microphone array apparatus M3 corresponding toselected marker M3R is selected.)

Thus, the operation switching control unit 38 can output the collectedsound to which different directivities are formed in the pluralomni-directional microphone array apparatuses M1, M2, M3 and M4associated with the selected camera apparatus C5 for the predeterminedtime. Thus, by performing a simple operation for selecting the collectedsound that is determined as optimal by the user, the operation switchingcontrol unit 38 can select the optimal omni-directional microphone arrayapparatus M3 capable of correctly collecting the sound emitted by themoving monitoring target (for example, the person HM1), and can collectthe sound emitted by the monitoring target (for example, the person HM1)with high accuracy.

Next, an automatic switching process of a camera apparatus in thedirectivity control system 100B according to the present embodiment willbe described with reference to FIGS. 29A. FIG. 29A is a flowchartillustrating an example of the automatic switching process of the cameraapparatus in the directivity control system 100B according to the secondembodiment. The automatic switching process of the camera apparatus inFIG. 29A describes contents of the automatic switching process of thecamera apparatus shown in FIG. 24 in detail, and is continuouslyperformed after step S3B-1 shown in FIG. 10B, for example.

In FIG. 29A, the image processing unit 37 performs a predetermined imageprocessing for image data projected onto the tracking screen TRW of thedisplay apparatus 35 to detect the position (i.e., the tracking positionA1) of the monitoring target (for example, the person HM1) (S21). Afterstep S21, a camera switching determination process is performed (S22).Details of the camera switching determination process will be describedlater with reference to FIG. 29B.

After step S22, if the camera switching mode is turned on in theoperation switching control unit 38 (S23, YES), the operation switchingcontrol unit 38 instructs all the switchable camera apparatusesassociated with the currently used camera apparatus (for example, thecamera apparatus C1) to capture an image through the communication unit31 and the network NW (S24). All the camera apparatuses that receive theimage capturing instruction start the image capturing. The cameraswitching mode indicates a flag used for control of process whether ornot to switch the camera apparatus when the plural-camera switchingmethod is set as an automatic operation.

The operation switching control unit 38 determines whether the personHM1 existing on the tracking position A1 in the actual space detected inthe step S21 is out of the imaging area C1RN of the camera apparatus C1using the information on the distance between the camera apparatus C1and the person HM1 measured by the currently used camera apparatus C1(S25). If it is determined that the person HM1 is out of the imagingarea C1RN of the camera apparatus C1 (S25, YES), the operation switchingcontrol unit 38 outputs the image data obtained by the imaging of allthe switchable camera apparatuses associated with the currently usedcamera apparatus C1 according to the instruction in step S24 to theimage processing unit 37. The image processing unit 37 performs apredetermined image processing for the entire image data output from theoperation switching control unit 38 to determine whether the person HM1that is the monitoring target is detected (S26). The image processingunit 37 outputs the image processing result to the operation switchingcontrol unit 38.

The operation switching control unit 38 can detect the person HM1 thatis the monitoring target using the image processing result of the imageprocessing unit 37. Further, the operation switching control unit 38selects one camera apparatus (for example, the camera apparatus C2)closest to the tracking position A1 in the actual space detected in thestep S21, and switches the camera apparatus used for capturing the imageof the person HM1 from the camera apparatus C1 to the camera apparatusC2 (S27). Thus, the output control unit 34 b switches the trackingscreen TRW displayed in the display apparatus 35 to the camera screen ofthe camera apparatus C2 selected by the operation switching control unit38 for display (S27).

On the other hand, if the camera switching mode is turned off by theoperation switching control unit 38 (S23, NO), or if it is determinedthat the person HM1 is out of the imaging area C1RN of the cameraapparatus C1 (S25, NO), the automatic switching process of the cameraapparatus shown in FIG. 29A is terminated, and then, the procedureproceeds to an automatic switching process of the omni-directionalmicrophone array apparatus shown in FIG. 30A.

Next, the camera switching determination process in the directivitycontrol apparatus 3B will be described with reference to FIG. 29B. FIG.29B is a flowchart illustrating an example of the camera switchingdetermination process shown in FIG. 29A.

In FIG. 29B, the operation switching control unit 38 turns off thecamera switching mode in the directivity control apparatus 3B (S22-1).The operation switching control unit 38 determines whether the trackingposition A1 in the actual space corresponding to the tracking pointdetected in step S21 is out of the predetermined switching determinationline JC1 of the camera apparatus C1 that is currently used, using theinformation on the distance between the camera apparatus C1 and theperson HM1 measured by the currently used camera apparatus C1 (S22-2).

If it is determined that the tracking position A1 in the actual spacecorresponding to the tracking point detected in the step S21 is out ofthe predetermined switching determination line JC1 of the currently usedcamera apparatus C1 (S22-2, YES), the operation switching control unit38 turns on (as an automatic mode) the camera switching mode (S22-3).

After step S22-3, or if it is determined that the tracking position A1is not out of the predetermined switching determination line JC1 of thecurrently used camera apparatus C1 (S22-2, NO), the camera switchingdetermination process shown in FIG. 29B is terminated, and then, theprocedure proceeds to step S23 shown in FIG. 29A.

Next, the automatic switching process in the omni-directional microphonearray apparatus in the directivity control system 100B according to thepresent embodiment will be described with reference to FIG. 30A. FIG.30A is a flowchart illustrating an example of the automatic switchingprocess of the omni-directional microphone array apparatus in thedirectivity control system 100B according to the second embodiment. Theautomatic switching process of the omni-directional microphone arrayapparatus shown in FIG. 30A describes the content of the automaticswitching process of the omni-directional microphone array apparatusshown in FIG. 25 in detail, and may be continuously performed after stepS27 shown in FIG. 29A, or reversely, the automatic switching process ofthe camera apparatus shown in FIG. 29A may be performed after theautomatic switching process of the omni-directional microphone arrayapparatus shown in FIG. 30A.

In FIG. 30A, the sound source detecting unit 34 d performs apredetermined sound source detection process to calculate the position(position of the sound source) of the monitoring target (for example,the person HM1) in the actual space, or to calculate coordinates (thatis, coordinates of the tracking position A1 corresponding to thetracking point) indicating a position on image data corresponding to thecalculated position of the sound source (S31). After step S31, amicrophone switching determination process is performed (S32). Detailsof the microphone switching determination process will be describedlater with reference to FIG. 30B.

After step S32, if the microphone switching mode is turned on by theoperation switching control unit 38 (S33, YES), the operation switchingcontrol unit 38 instructs all the switchable omni-directional microphonearray apparatuses associated with the currently used omni-directionalmicrophone array apparatus (for example, the omni-directional microphonearray apparatus M1) to collect the sound emitted by the person HM1through the communication unit 31 and the network NW (S34). All theomni-directional microphone array apparatuses that receive the soundcollection instruction start the sound collection. The camera switchingmode indicates a flag used for control of process whether or not toswitch the camera apparatus when the plural-camera switching method isset as an automatic operation.

The operation switching control unit 38 determines whether the personHM1 is out of the sound collecting area M1RN of the omni-directionalmicrophone array apparatus M1 using the information on the distancebetween the currently used omni-directional microphone array apparatusM1 and the person HM1 calculated by the sound source detecting unit 34 d(S35). If it is determined that the person HM1 is out of the soundcollecting area M1RN of the omni-directional microphone array apparatusM1 (S35, YES), the sound source detecting unit 34 d calculates theposition (position of the sound source) of the person HM1 that is themonitoring target based on the strength or the sound volume level of thesound collected by all the switchable omni-directional microphone arrayapparatuses M1 associated with the currently used omni-directionalmicrophone array apparatus according to the instruction in step S34(S36).

The operation switching control unit 38 selects one omni-directionalmicrophone array apparatus (for example, the omni-directional microphonearray apparatus M2) in which the distance difference between theposition (position of the sound source) of the person HM1 that is themonitoring target and the omni-directional microphone array apparatus isthe minimum, among all the omni-directional microphone array apparatusesassociated with the currently used omni-directional microphone arrayapparatus M1 using the sound source detection result of the sound sourcedetecting unit 34 d, and switches the omni-directional microphone arrayapparatus used for the sound collection emitted by the person HM1 fromthe omni-directional microphone array apparatus M1 to theomni-directional microphone array apparatus M2 (S37). Thus, the outputcontrol unit 34 b switches the directivity of the sound in a directionfrom the switched omni-directional microphone array apparatus M2 towardthe position of the sound source calculated in step S36 (S37).

On the other hand, if the microphone switching mode is turned off by theoperation switching control unit 38 (S33, NO), or if it is determinedthat the person HM1 is not out of the sound collecting area M1RN of theomni-directional microphone array apparatus M1 (S35, NO), the automaticswitching process of the omni-directional microphone array apparatusshown in FIG. 30A is terminated, and then, the procedure proceeds tostep S3B-2 shown in FIG. 10B. The automatic switching process of thecamera apparatus shown in FIG. 29A may be started after the automaticswitching process of the omni-directional microphone array apparatusshown in FIG. 30A is terminated.

Next, the microphone switching determination process in the directivitycontrol apparatus 3B will be described with reference to FIG. 30B. FIG.30B is a flowchart illustrating an example of the microphone switchingdetermination process shown in FIG. 30A.

In FIG. 30B, the operation switching control unit 38 turns off themicrophone switching mode (S32-1). The operation switching control unit38 determines whether the tracking position A1 calculated in step S31 isout of the predetermined switching determination line JM1 of theomni-directional microphone array apparatus M1 that is currently used,using the information on the distance between the currently usedomni-directional microphone array apparatus M1 and the person HM1(S32-2).

If it is determined that the tracking position A1 is out of thepredetermined switching determination line JM1 of the currently usedomni-directional microphone array apparatus M1 (S32-2, YES), theoperation switching control unit 38 turns on the microphone switchingmode (S32-3).

After step S32-3, or if it is determined that the tracking position A1is not out of the predetermined switching determination line JM1 of thecurrently used omni-directional microphone array apparatus M1 (S32-2,NO), the microphone switching determination process shown in FIG. 30B isterminated, and then, the procedure proceeds to step S33 shown in FIG.30A.

Next, a manual switching process of the camera apparatus in thedirectivity control system 100B according to the present embodiment willbe described with reference to FIG. 31A. FIG. 31A is a flowchartillustrating an example of the manual switching process of the cameraapparatus in the directivity control system 100B according to the secondembodiment. The manual switching process of the camera apparatus in thedirectivity control system 100B shown in FIG. 31A is performedsubsequently to the step S1 shown in FIG. 9A, FIG. 9B or FIG. 10A.

In FIG. 31A, if an instruction for switching the camera apparatus isinput to the display apparatus 35 according to an input operation usingthe cursor CSR using the mouse operation of the user or the finger FG ofthe user (S41), the output control unit 34 b switches the trackingscreen TRW of the image obtained by the imaging of the camera apparatusC1 currently used for capturing the image of the person HM1 to themulti-camera screen that includes the camera screen C1W of the cameraapparatus C1 and the camera screens of the camera apparatuses (forexample, eight camera apparatuses) around the camera apparatus C1 (S42).

It is assumed that the user selects any camera screen by the touchoperation of the finger FG, for example, with respect to themulti-camera screen displayed in the display apparatus 35 in step S42,in consideration of the movement direction MV1 of the person HM1 that isthe monitoring target (see FIG. 26) (S43).

The operation switching control unit 38 switches the camera apparatusused for capturing the image of the person HM1 from the currently usedcamera apparatus C1 to the camera apparatus C3 corresponding to thecamera screen C3W that is the target of the touch operation in step S43,according to the touch operation of the finger FG of the user (S44).Thus, the manual switching process of the camera apparatus shown in FIG.31A is terminated, and then, the procedure proceeds to either one ofsteps S45, S51, S61 and step S71 shown in FIG. 31B, FIG. 32A or FIG.32B.

Next, the manual switching process of the omni-directional microphonearray apparatus in the directivity control system 100B according to thepresent embodiment will be described with reference to FIG. 31B. FIG.31B is a flowchart illustrating an example of the manual switchingprocess of the omni-directional microphone array apparatus in thedirectivity control system 100B according to the second embodiment.

In FIG. 31B, if an instruction for switching the omni-directionalmicrophone array apparatus according to an input operation using thecursor CSR using the mouse operation of the user or the finger FG of theuser (S45), the output control unit 34 b displays markers (for example,markers M2R, M3R and M4R) indicating rough positions of theomni-directional microphone array apparatuses (for example, theomni-directional microphone array apparatuses M2, M3 and M4) capable ofbeing switched from the currently used omni-directional microphone arrayapparatus M1 on the tracking screen TRW (S46).

The user selects any one marker (for example, the marker M3R) from thethree markers by the touch operation of the finger FG of the user inconsideration of the movement direction MV1 from the tracking positionA1 of the person HM1 that is the monitoring target (S47, see FIG. 27).The operation switching control unit 38 instructs the omni-directionalmicrophone array apparatus M3 corresponding to the marker M3R selectedby the touch operation of the finger FG of the user, instead of thecurrently used omni-directional microphone array apparatus M1, to startthe sound collection through the communication unit 31 and the networkNW (S47).

The output control unit 34 b switches the directivity in a directionfrom the omni-directional microphone array apparatus M3 corresponding tothe marker M3R selected in step S47 toward the tracking position of theperson HM1 at the current time (S48). Further, the output control unit34 b deletes the markers M2R, M3R and M4R indicating the rough positionsof the omni-directional microphone array apparatuses M2, M3 and M4displayed on the tracking screen TRW (S48).

After step S48, the manual switching process of the omni-directionalmicrophone array apparatus shown in FIG. 31B is terminated, and then,the procedure proceeds to step S2 shown in FIG. 9A, FIG. 9B or FIG. 10A.Here, after the manual switching process of the omni-directionalmicrophone array apparatus shown in FIG. 31B, the manual switchingprocess of the camera apparatus shown in FIG. 31A may be performed.

Next, a selection process of the omni-directional microphone arrayapparatus optimal in the directivity control system 100B according tothe present embodiment will be described with reference to FIGS. 32A and32B and FIG. 33. FIG. 32A is a flowchart illustrating a first example ofthe selection process of the omni-directional microphone array apparatusoptimal in the directivity control system 100B according to the secondembodiment. FIG. 32B is a flowchart illustrating a second example of theselection process of the omni-directional microphone array apparatusoptimal in the directivity control system 100B according to the secondembodiment. FIG. 33 is a flowchart illustrating a third example of theselection process of the omni-directional microphone array apparatusoptimal in the directivity control system 100B according to the secondembodiment.

In FIG. 32A, if the position (the tracking position corresponding to thetracking point) of the person HM1 that is the monitoring target in themovement direction is designated on the tracking screen TRW displayed inthe display apparatus 35 according to the input operation based on thecursor CSR using the mouse operation of the user or the finger FG of theuser (S51), information on the designated position (for example,coordinates) is input to the operation switching control unit 38 (S52).

The operation switching control unit 38 calculates individual distancefrom each omni-directional microphone array apparatus to the position inthe actual space corresponding to the designated position designated instep S51, that is, individual distance from each omni-directionalmicrophone array apparatus to the person HM1 that is the monitoringtarget (S53).

The operation switching control unit 38 selects the omni-directionalmicrophone array apparatus having the minimum distance among therespective distances calculated in step S53, and instructs the signalprocessing unit 34 to form the directivity with respect to sound data ofthe sound collected by the selected omni-directional microphone arrayapparatus (S54).

The output control unit 34 b of the signal processing unit 34 forms thedirectivity of the sound in a direction from the omni-directionalmicrophone array apparatus selected by the operation switching controlunit 38 in step S54 toward the position of the person HM1 that is themonitoring target according to the instruction in step S54, and outputsthe sound to which the directivity is formed through the speaker 36(S55).

Thus, as the user easily designates the position indicating the movementdirection of the monitoring target (for example, the person HM1), theoperation switching control unit 38 can select the optimalomni-directional microphone array apparatus capable of correctlycollecting the sound emitted by the monitoring target (for example, theperson HM1) during movement, and can collect the sound emitted by themonitoring target (for example, the person Hm1 with high accuracy.

After step S55, the selection process of the optimal omni-directionalmicrophone array apparatus shown in FIG. 32A is terminated, and then,the procedure proceeds to step S2 shown in FIG. 9A, FIG. 9B or FIG. 10A.Here, after the optimal omni-directional microphone array apparatusselection process shown in FIG. 32A, the manual switching process shownin FIG. 31A may be performed.

In FIG. 32B, if the position (the tracking position corresponding to thetracking point) of the person HM1 that is the monitoring target in themovement direction is designated on the tracking screen TRW displayed inthe display apparatus 35 according to the input operation based on thecursor CSR using the mouse operation of the user or the finger FG of theuser (S61), information on the designated position (for example,coordinates) is input to the operation switching control unit 38.

Thus, by performing a predetermined image processing for the image datacaptured by the currently used camera apparatus (for example, the cameraapparatus C1), the image processing unit 37 detects the direction of theface of the person HM1 that is the monitoring target (S62). The imageprocessing unit 37 outputs the detection result of the direction of theface of the person HM1 that is the monitoring target to the operationswitching control unit 38.

The operation switching control unit 38 calculates the relationshipbetween the face of the person HM1, the designated position and eachomni-directional microphone array apparatus using the information on thedesignated position designated in step S61 (for example, coordinatesindicating the position on the image data) and the detection result ofthe direction of the face of the person HM1 obtained by the imageprocessing unit 37 in step S62 (S63). For example, the operationswitching control unit 38 calculates the distance between the positionof the monitoring target (for example, the person HM1) corresponding tothe designated position on the image data designated in step S61 andeach omni-directional microphone array apparatus.

The operation switching control unit 38 selects the omni-directionalmicrophone array apparatus having the minimum distance between theposition of the monitoring target (for example, the person HM1)corresponding to the designated position on the image data designated instep S61 and each omni-directional microphone array apparatus, in adirection (for example, within 45 degrees in the horizontal direction)along the direction of the face of the monitoring target (for example,the person HM1) (S64). Further, the operation switching control unit 38instructs the signal processing unit 34 to form the directivity withrespect to the sound data on the sound collected by the omni-directionalmicrophone array apparatus selected in step S64 (S64).

The output control unit 34 b of the signal processing unit 34 forms thedirectivity of the sound in a direction from the omni-directionalmicrophone array apparatus selected in step S64 toward the position ofthe person HM1 that is the monitoring target according to theinstruction in step S64, and outputs the sound to which the directivityis formed from the speaker 36 (S65).

Thus, the operation switching control unit 38 can select the optimalomni-directional microphone array apparatus capable of correctlycollecting the sound emitted by the monitoring target (for example, theperson HM1) during movement according to the direction of the face onthe image data of the monitoring target (for example, the person HM1)and the distance between the monitoring target (for example, the personHM1) and each omni-directional microphone array apparatus, and cancollect the sound emitted by the monitoring target (for example, theperson HM1) with high accuracy.

After step S65, the selection process of the optimal omni-directionalmicrophone array apparatus shown in FIG. 32B is terminated, and then,the procedure proceeds to step S2 shown in FIG. 9A, FIG. 9B or FIG. 10A.Here, after the optimal omni-directional microphone array apparatusselection process shown in FIG. 32B, the manual switching process of thecamera apparatuses shown in FIG. 31A may be performed.

In FIG. 33, the output control unit 34 b displays in list the camerascreens of all the camera apparatuses controlled by the directivitycontrol system 100B in the display apparatus 35 according to an inputoperation based on the cursor CSR using the mouse operation of the useror the finger FG of the user (S71). It is assumed that on the camerascreen on which the monitoring target (for example, the person HM1) thatis the sound tracking process target is projected among the respectivecamera screens that are displayed in list in the display apparatus 35,the camera screen C1W where the projection of the person HM1 is the mostexcellent is selected according to an input operation based on thecursor CSR using the mouse operation of the user or the finger FG of theuser (S72).

The operation switching control unit 38 selects the camera apparatuscorresponding to the camera screen as the camera apparatus used forcapturing the image of the person HM1 according to the selection of thecamera screen of the user in step S72, for switching. Thus, the outputcontrol unit 34 b enlarges the image data obtained by the imaging of thecamera apparatus corresponding to the camera screen, and displays theresult on the tracking screen TRW1 of the display apparatus 35 (S73, seea lower left side in FIG. 28).

Further, the output control unit 34 b displays the markers (for example,the markers M1R, M2R, M3R and M4R shown in FIG. 28) indicating roughpositions of all the omni-directional microphone array apparatusesassociated with the camera apparatus selected by the operation switchingcontrol unit 38 at four corners of the tracking screen TRW1 (S74).

The markers M1R, M2R, M3R and M4R are sequentially designated accordingto an input operation based on the cursor CSR using the mouse operationof the user or the finger FG of the user (S75), the output control unit34 b forms the directivity in a direction from the omni-directionalmicrophone array apparatus corresponding to each marker to the positionof the person HM1 while highlighting (for example, by a blink Br) themarkers one by one, and outputs the sound collected for a predeterminedtime (S76).

If the marker (for example, the marker M3R) indicating the roughposition of the omni-directional microphone array apparatus that isdetermined as optimal by the user in the sound output for apredetermined time is selected, the operation switching control unit 38selects the omni-directional microphone array apparatus M3 correspondingto the selected marker M3R as the omni-directional microphone arrayapparatus used for collection of the sound emitted by the person HM1,for switching (S77).

After step S77, the optimal omni-directional microphone array apparatusselection process shown in FIG. 33 is terminated, and then, theprocedure proceeds to step S2 shown in FIG. 9A, FIG. 9B or FIG. 10A.Here, after the optimal omni-directional microphone array apparatusselection process shown in FIG. 33, the manual switching process of thecamera apparatus shown in FIG. 31A may be performed.

(Modification Example of the First Embodiment)

In the above-described respective embodiments, when a single monitoringtarget (for example, the person HM1) is mainly projected onto the imagedata, the sound tracking process based on the movement of the person HM1that is the single monitoring target is performed.

In a modification example (hereinafter, referred to as “the presentmodification example”) of the first embodiment, when plural monitoringtargets (for example, plural persons) appear on the tracking screen TRWin the first embodiment or the second embodiment, an operation exampleof the directivity control system 100 when the plural persons aredesignated at the same timing or different timings will be described.Since a system configuration example of the directivity control systemof the present modification example is the same as in the directivitycontrol system 100, 100A or 100B according to the first embodiment orthe second embodiment, the description of the system configurationexample will be simplified or not be repeated, and only the contentsthat are different will be described. Hereinafter, description will beprovided with reference to the system configuration example of thedirectivity control system 100 for ease of description.

The operation example of the directivity control system 100 according tothe present modification example will be described with reference toFIGS. 34 and 35. FIG. 34 is a flowchart illustrating an example of anoverall flow of a manual tracking process based on plural simultaneousdesignations in the directivity control system 100 according to themodification example of the first embodiment. FIG. 35 is a flowchartillustrating an example of an automatic tracking process of pluralmonitoring targets in the directivity control system 100 according tothe modification example of the first embodiment. In FIG. 35, thedirectivity control apparatuses 3A and 3B are used.

In FIG. 34, since the determination process of a tracking mode in stepS1, the tracking auxiliary process in step S2, the tracking connectionprocess in step S5 and the sound output process in step S7 respectivelycorrespond to the determination process of a tracking mode in step S1,the tracking auxiliary process in step S2 shown in FIG. 9A, the trackingconnection process in step S6 shown in FIG. 9A and the sound outputprocess in step S7 shown in FIG. 9A, for example, description thereofwill not be repeated.

In FIG. 34, if the tracking mode is turned off (S1, NO), the manualtracking process based on the plural simultaneous designations shown inFIG. 34 is terminated, but if the tracking mode is turned on (S1, YES),on the tracking screen TRW of the display apparatus 35, the sound thatis currently output (reproduced) from the speaker 36 is temporarilystopped according to a click operation of the cursor CSR using the mouseoperation of the user or a touch operation using the finger FG of theuser (S81). After step S81, the tracking auxiliary process is performed(S2).

After step S2, the tracking positions of the movement courses (movementpaths) of the plural persons that are the monitoring targets, that is,the plural tracking points are simultaneously designated according to aninput operation based on the cursor CSR using the mouse operation of theuser or the finger FG of the user (S82).

The tracking processing unit 34 c distinguishes the positions in theactual space corresponding to the plural designated positions on thetracking screen TRW and the designation times for each person that isthe monitoring target designated in step S82 and stores the designatedposition and the designation time in the memory 33 in association as thetracking position and the tracking time of each tracking point (S83).Further, the tracking processing unit 34 c distinctly displays the pointmarker on the tracking point on the tracking screen TRW for each personthat is the monitoring target through the output control unit 34 b(S85).

The output control unit 34 b forms the directivity of the collectionsound in a direction from the currently used omni-directional microphonearray apparatus (for example, omni-directional microphone arrayapparatus) M1 toward the actual position (sound position or sound sourceposition) of each person corresponding to the tracking position for eachof the persons that are the plural monitoring targets that aresimultaneously designated in step S82 (S84). After step S84, thetracking connection process is performed (S6).

After step S6, the output control unit 34 b restarts the output(reproduction) of the sound that is temporarily stopped in step S81through the speaker 36 (S85). Further, after step S85, the sound outputprocess is performed (S7). After step S7, the operations from step S81through step S7 (i.e., the operations of steps S81, S2, S82, S83, S84,S6, S85 and S7) are repeated until the tracking mode of the directivitycontrol apparatus 3B is turned off.

In FIG. 35, after step S83, the image processing unit 37 of thedirectivity control apparatuses 3A and 3B determines whether the personthat is the monitoring target is detected on the tracking screen TRW ofthe display apparatus 35 by performing a known image processing. If itis determined that plural persons are detected, the image processingunit 37 outputs the determination result (including data on the detectedposition (for example, a known representative point) of each person andthe detection time) to the tracking processing unit 34 c of the signalprocessing unit 34 (S91) as an automatic designation result. Further,the sound source detecting unit 34 d determines whether the position ofthe sound (sound source) emitted by the person that is the monitoringtarget is detected on the tracking screen TRW of the display apparatus35 by performing a known sound source detection process. If it isdetermined that positions of plural sound sources are detected, thesound source detecting unit 34 d outputs the determination result(including data on the detected position and the detection time of thesound) to the tracking processing unit 34 c as an automatic designationresult (S91).

The tracking processing unit 34 c calculates a motion vector of each ofthe persons that are the plural monitoring targets using the change ofone or more last automatic designation results in step S91, andestimates a movement direction of each person (S91).

The tracking processing unit 34 c stores the tracking position and eachprevious automatic designation result corresponding to the pluraltracking points automatically designated in the memory 33 in associationas a pair for the tracking position using the estimation result of themovement directions of the persons that are the plural monitoringtargets in step S91 (S92). The tracking processing unit 34 cdistinguishes the plural designated positions of the respective personson the tracking screen TRW and the designation times for each personthat is the monitoring target, and stores the designated position andthe designation time in the memory 33 in association as the trackingposition and the tracking time of each tracking point (S92). Further,the tracking processing unit 34 c distinctly displays the point markeron the tracking position on the tracking screen TRW for each person thatis the monitoring target through the output control unit 34 b (S92).

Thus, even though the plural monitoring targets (for example, persons)projected onto the image data on the tracking screen TRW of the displayapparatus 35 move in what way, the directivity control apparatuses 3, 3Aand 3B of the present modification example form the directivity of thesound formed in a direction toward the position of each person beforemovement in a direction toward the position of each person aftermovement, and thus, can appropriately form the directivity of the soundin a tracking manner according to the movement of each person, and cansuppress deterioration of monitoring work efficiency of an observer.

Hereinafter, a description is made of examples of configurations,actions and advantages of a directivity control apparatus, a directivitycontrol method, a storage medium and a directivity control systemaccording to aspects of the present invention.

A first aspect of the present invention provides a directivity controlapparatus for controlling a directivity of a sound collected by a firstsound collecting unit including a plurality of microphones, thedirectivity control apparatus including: a directivity forming unit,configured to form a directivity of the sound in a direction toward amonitoring target corresponding to a first designated position in animage displayed on a display unit; and an information obtaining unit,configured to obtain information on a second designated position in theimage displayed on the display unit, designated in accordance with amovement of the monitoring target, wherein the directivity forming unitis configured to change the directivity of the sound toward themonitoring target corresponding to the second designated position byreferring to the information on the second designated position obtainedby the information obtaining unit.

The directivity control apparatus may be configured so that theinformation obtaining unit is configured to obtain the information onthe second designated position in accordance with a designatingoperation to the monitoring target which moves in the image displayed onthe display unit.

The directivity control apparatus may be configured by further includinga sound source detecting unit, configured to detect a position of asound source corresponding to the monitoring target from the imagedisplayed on the display unit; and an image processing unit, configuredto detect the monitoring target from the image displayed on the displayunit, wherein the information obtaining unit is configured to obtain, asthe information on the second designated position, information on theposition of the sound source detected by the sound source detecting unitor information on a position of the monitoring target detected by theimage processing unit.

The directivity control apparatus may be configured so that the soundsource detecting unit is configured to start a detection processing ofthe position of the sound source corresponding to the monitoring targetwhile placing an initial position designated in the image displayed onthe display unit at a center of the detection processing, and the imageprocessing unit is configured to start a detection processing of themonitoring target while placing the initial position at a center of thedetection processing.

The directivity control apparatus may be configured so that theinformation obtaining unit is configured to obtain, as the informationon the second designated position, information on a position designatedby a changing operation in the image displayed on the display unit inaccordance with the changing operation, wherein the changing operationindicates an operation for changing the information on the position ofthe sound source detected by the sound source detecting unit or theinformation on the position of the monitoring target detected by theimage processing unit.

The directivity control apparatus may be configured so that theinformation obtaining unit is configured to obtain, as the informationon the second designated position, information on a position designatedby a changing operation in the image displayed on the display unit inaccordance with the changing operation, in a case where a distancebetween the position of the sound source detected by the sound sourcedetecting unit and the position of the monitoring target detected by theimage processing unit is equal to or larger than a predetermineddistance value, wherein the changing operation indicates an operationfor changing the information on the position of the sound source or theinformation on the position of the monitoring target.

The directivity control apparatus may be configured by furtherincluding: an image storage unit that stores images captured for a givenperiod of time; and an image reproduction unit, configured to reproducethe images stored in the image storage unit on the display unit, whereinthe image reproduction unit is configured to reproduce the images at aspeed value which is smaller than an initial value of a reproductionspeed in accordance with a predetermined input operation.

The directivity control apparatus may be configured by furtherincluding: a display control unit, configured to display a capturedimage on the display unit, wherein the display control unit isconfigured to enlarge and display the image on a same screen at apredetermined magnification in accordance with a designation of adesignated position in the image displayed on the display unit whileplacing the designated position at a center of enlargement.

The directivity control apparatus may be configured by furtherincluding: a display control unit, configured to display a capturedimage on the display unit, wherein the display control unit isconfigured to enlarge and display the image on a different screen at apredetermined magnification in accordance with a designation of adesignated position in the image displayed on the display unit whileplacing the designated position at a center of enlargement.

The directivity control apparatus may be configured by furtherincluding: a display control unit, configured to display a capturedimage on the display unit, wherein the display control unit isconfigured to enlarge and display the image at a predeterminedmagnification with respect to a center of the display unit.

The directivity control apparatus may be configured so that the displaycontrol unit is configured to scroll by a predetermined amount anenlarged screen in which the image is enlarged and displayed in adirection out of a scroll determination line in a case where thedesignated position is out of the scroll determination line in theenlarged screen in accordance with a movement of the monitoring target.

The directivity control apparatus may be configured so that the displaycontrol unit is configured to scroll an enlarged screen in which theimage is enlarged and displayed while centering the second designatedposition in a case where the designated position is out of a scrolldetermination line in the enlarged screen in accordance with a movementof the monitoring target.

The directivity control apparatus may be configured so that the displaycontrol unit is configured to scroll an enlarged screen in which theimage is enlarged and displayed while placing the designated position ata center of the enlarged screen.

The directivity control apparatus may be configured so that the imageprocessing unit is configured to perform a masking process of partlymasking on the monitoring target in the image displayed on the displayunit.

The directivity control apparatus may be configured by furtherincluding: a sound output control unit, configured to cause a soundoutput unit to output the sound collected by the first sound collectingunit, wherein the sound output control unit is configured to cause thesound output unit to perform a voice change process on the soundcollected by the first sound collecting unit and cause the sound outputunit to output the processed sound.

The directivity control apparatus may be configured by furtherincluding: a sound storage unit that stores a sound collected by thefirst sound collecting unit for a given period of time; and a soundoutput control unit, configured to cause a sound output unit to outputthe sound stored in the sound storage unit, wherein the sound outputcontrol unit is configured to cause the sound output unit to perform avoice change process on the sound stored in the sound storage unit andcause the sound output unit to output the processed sound.

The directivity control apparatus may be configured by furtherincluding: a display control unit, configured to display a predeterminedmarker on each of one or more designated positions designated inaccordance with a movement of the monitoring target in the imagedisplayed on the display unit.

The directivity control apparatus may be configured by furtherincluding: a display control unit, configured to display a connectingline at least between a current designated position and a lastdesignated position among two or more designated positions designated inaccordance with a movement of the monitoring target in the imagedisplayed on the display unit.

The directivity control apparatus may be configured by furtherincluding: a display control unit, configured to display a movementroute connecting between each designated position and one or twoadjacent designated positions for all designated positions designated inaccordance with a movement of the monitoring target in the imagedisplayed on the display unit.

The directivity control apparatus may be configured by furtherincluding: a designation list storage unit that stores a designationlist that includes data on the all designated positions in the imagedisplayed on the display unit and designated times; and a reproductiontime calculating unit, configured to calculate a reproduction startingtime of the sound in an arbitrary designated position on a movementroute connecting between the all designated positions displayed by thedisplay control unit in accordance with a designation of the arbitrarydesignated position on the movement route by referring to thedesignation list stored in the designation list storage unit, whereinthe directivity forming unit is configured to form the directivity ofthe sound by using the data on the designated position corresponding tothe designated time closest to the reproduction starting time of thesound calculated by the reproduction time calculating unit.

The directivity control apparatus may be configured by furtherincluding: a sound storage unit that stores a sound collected by thefirst sound collecting unit for a given period of time; and a soundoutput control unit, configured to cause a sound output unit to outputthe sound stored in the sound storage unit, wherein the sound outputcontrol unit is configured to cause the sound output unit to output thesound at the reproduction starting time of the sound calculated by thereproduction time calculating unit, and the directivity forming unit isconfigured to form the directivity of the sound by using the data on thedesignated position corresponding to a subsequent designated time whenthe subsequent designated time exists in a predetermined time of periodfrom the reproduction starting time of the sound.

The directivity control apparatus may be configured by furtherincluding: an operation switching control unit, configured to switchimaging for display of the image on the display unit from a firstimaging unit to a second imaging unit in a case where the monitoringtarget is out of a predetermined switching area corresponding to thefirst imaging unit.

The directivity control apparatus may be configured by furtherincluding: an operation switching control unit, configured to switchcollecting of the sound from the monitoring target from the first soundcollecting unit to a second sound collecting unit in a case where themonitoring target is out of a predetermined switching area correspondingto the first sound collecting unit.

The directivity control apparatus may be configured by furtherincluding: a display control unit, configured to display in a listimages captured by a plurality of imaging units in different screens onthe display unit in accordance with a predetermined input operation; andan operation switching control unit, configured to select an imagingunit for displaying an image of the monitoring target on the displayunit in accordance with an operation of selecting any one of selectablescreens among the screens displayed in the list on the display unit bythe display control unit.

The directivity control apparatus may be configured by furtherincluding: a display control unit, configured to display markers roughlyindicating positions of a plurality of sound collecting units disposedaround the first sound collecting unit and switchable from the firstsound collecting unit in accordance with a predetermined inputoperation; and an operation switching control unit, configured to switchcollecting of the sound from the monitoring target from the first soundcollecting unit to another sound collecting unit corresponding to aselected marker in accordance with an operation of selecting the markerfrom the markers displayed on the display unit by the display controlunit.

The directivity control apparatus may be configured so that theoperation switching control unit is configured to select one soundcollecting unit for collecting the sound from the monitoring target,which is closest to the monitoring target from a designated position ofthe monitoring target in the image captured by the imaging unit selectedby the operation switching control unit, among a plurality of soundcollecting units including the first sound collecting unit in accordancewith a designation of the position.

The directivity control apparatus may be configured by furtherincluding: an image processing unit, configured to detect a direction ofa face of the monitoring target from the image on the display unit,wherein the operation switching control unit is configured to select asound collecting unit for collecting the sound from the monitoringtarget, which is closest to the monitoring target from a designatedposition of the monitoring target in the image captured by the imagingunit selected by the operation switching control unit in a directioncorresponding to the direction of the face of the monitoring targetdetected by the image processing unit, among a plurality of soundcollecting units including the first sound collecting unit in accordancewith a designation of the position.

The directivity control apparatus may be configured by furtherincluding: a sound output control unit, configured to cause a soundoutput unit to output the sound collected by the first sound collectingunit, wherein the display control unit is configured to display on thedisplay unit markers roughly indicating positions of a plurality ofsound collecting units including the first sound collecting unitcorresponding to the imaging unit selected by the operation switchingcontrol unit, the sound output control unit is configured tosequentially output for a predetermined time of period a sound includinga directivity formed in a direction of the monitoring target from eachof the plurality of sound collecting units corresponding to respectiveof the markers displayed on the display unit in accordance with adesignation of a position of the monitoring target in the image capturedby the imaging unit selected by the operation switching control unit,and the operation switching control unit is configured to select a soundcollecting unit for collecting the sound from the monitoring target,which corresponds to a selected marker in accordance with an operationof selecting the marker based on the sound output by the sound outputcontrol unit.

A second aspect of the present invention provides a directivity controlmethod in a directivity control apparatus for controlling a directivityof a sound collected by a first sound collecting unit including aplurality of microphones, the directivity control method including:forming a directivity of the sound in a direction toward a monitoringtarget corresponding to a first designated position in an imagedisplayed on a display unit; obtaining information on a seconddesignated position in the image displayed on the display unit,designated in accordance with a movement of the monitoring target; andchanging the directivity of the sound toward the monitoring targetcorresponding to the second designated position by referring to theinformation on the second designated position.

A third aspect of the present invention provides a storage medium inwhich a program is stored, the program causing a computer in adirectivity control apparatus for controlling a directivity of a soundcollected by a first sound collecting unit including a plurality ofmicrophones, to execute: forming a directivity of the sound in adirection toward a monitoring target corresponding to a first designatedposition in an image displayed on a display unit; obtaining informationon a second designated position in the image displayed on the displayunit, designated in accordance with a movement of the monitoring target;and changing the directivity of the sound toward the monitoring targetcorresponding to the second designated position by referring to theinformation on the second designated position.

A fourth aspect of the present invention provides a directivity controlsystem including: an imaging unit, configured to capture an image in asound collecting area; a first sound collecting unit including aplurality of microphones, configured to collect a sound in the soundcollecting area; and a directivity control apparatus, configured tocontrol a directivity of the sound collected by the first soundcollecting unit, wherein the directivity control apparatus includes: adirectivity forming unit, configured to form a directivity of the soundin a direction toward a monitoring target corresponding to a firstdesignated position in an image displayed on a display unit; and aninformation obtaining unit, configured to obtain information on a seconddesignated position in the image displayed on the display unit,designated in accordance with a movement of the monitoring target,wherein the directivity forming unit changes the directivity of thesound toward the monitoring target corresponding to the seconddesignated position by referring to the information on the seconddesignated position obtained by the information obtaining unit.

Hereinbefore, various embodiments have been described with reference tothe accompanying drawings, but the present invention is not limited tothe embodiments as described. It is obvious that various modificationsor corrections can be made by those skilled in the art within the scopeof the present invention and understood that those modifications andcorrections belong to the technical range of the present invention.

The present invention is useful for a directivity control apparatus, adirectivity control method, a storage medium and a directivity controlsystem that appropriately form directivity of a sound with respect to amonitoring target in a tracking manner and prevent deterioration ofmonitoring efficiency of an observer even when the monitoring target onan image moves.

1-20. (canceled)
 21. A directivity control method of controlling adirectivity of a sound collected by a first sound collector, including aplurality of microphones, the directivity control method comprising:displaying a captured image on a display; enlarging the captured imageand displaying an enlarged image on the display; forming a directivityof the sound in a direction toward a monitoring target corresponding toa first designated position in the image displayed on the display;obtaining information on a second designated position in the imagedisplayed on the display, designated in accordance with a movement ofthe monitoring target; and changing the directivity of the sound towardthe monitoring target corresponding to the second designated position byreferring to the information on the second designated position.
 22. Thedirectivity control method according to claim 21, wherein in theobtaining, the information on the second designated position is obtainedin accordance with a designating operation of the monitoring targetwhich moves in the image displayed on the display.
 23. The directivitycontrol method according to claim 21, further comprising: detecting aposition of a sound source corresponding to the monitoring target fromthe image displayed on the display; and detecting the monitoring targetfrom the image displayed on the display, wherein in the obtaining,information on the detected position of the sound source or informationon a position of the detected monitoring target is obtained as theinformation on the second designated position.
 24. The directivitycontrol method according to claim 23, wherein in the sound sourceposition detecting, detection processing of the position of the soundsource corresponding to the monitoring target is started while placingan initial position designated in the image displayed on the display ata center of an area on which the detection processing of the position ofthe sound source is performed, and in the monitoring target detection,detection processing of the monitoring target is started while placingthe initial position at a center of an area on which the detectionprocessing of the monitoring target is performed.
 25. The directivitycontrol method according to claim 23, wherein the obtaining obtains, asthe information on the second designated position, information on aposition designated by a changing operation in the image displayed onthe display in accordance with the changing operation, wherein thechanging operation indicates an operation of changing the information onthe detected position of the sound source or the information on theposition of the detected monitoring target.
 26. The directivity controlmethod according to claim 23, wherein the obtaining obtains, as theinformation on the second designated position, information on a positiondesignated by a changing operation in the image displayed on the displayin accordance with the changing operation, when a distance between thedetected position of the sound source and the position of the detectedmonitoring target is equal to or larger than a predetermined distance,wherein the changing operation indicates an operation of changing theinformation on the detected position of the sound source or theinformation on the position of the detected monitoring target.
 27. Thedirectivity control method according to claim 21, further comprising:storing, in an image storage, images captured for a predetermined periodof time; and reproducing the images stored in the image storage on thedisplay, wherein in the reproducing, the images are reproduced at aspeed which is smaller than an initial reproduction speed, in accordancewith a predetermined input operation.
 28. The directivity control methodaccording to claim 21, further comprising: enlarging and displaying theimage at a predetermined magnification with respect to a center of thedisplay.
 29. The directivity control method according to claim 21,further comprising: scrolling the screen, in which the image is enlargedand displayed, when the second designated position is out of a scrolldetermination line in the screen, in accordance with a movement of themonitoring target.
 30. The directivity control method according to claim21, further comprising: scrolling the screen, in which the image isenlarged and displayed, while placing the second designated position ata center of the screen, when the second designated position is out of ascroll determination line in the screen in accordance with a movement ofthe monitoring target.
 31. The directivity control method according toclaim 21, further comprising: scrolling the screen, in which the imageis enlarged and displayed, while placing the second designated positionat a center of the screen.
 32. The directivity control method accordingto claim 23, wherein in the detecting of the monitoring target, amasking process of partly masking the monitoring target in the imagedisplayed on the display, is performed.
 33. The directivity controlmethod according to claim 21, further comprising: causing a sound outputdevice to perform a voice change process on the sound collected by thefirst sound collector; and causing the sound output device to output theprocessed sound.
 34. The directivity control method according to claim21, further comprising: storing, in a sound storage, a sound collectedby the first sound collector for a predetermined period of time; causinga sound output device to perform a voice change process on the soundstored in the sound storage; and causing the sound output device tooutput the processed sound.
 35. The directivity control method accordingto claim 21, further comprising: displaying a predetermined marker oneach of at least one designated position that is designated inaccordance with a movement of the monitoring target in the imagedisplayed on the display.
 36. The directivity control method accordingto claim 21, further comprising: switching an image for display from afirst imager to a second imager, when the monitoring target is out of apredetermined switching area corresponding to the first imager.
 37. Thedirectivity control method according to claim 21, further comprising:switching collecting of the sound from the monitoring target from thefirst sound collector to a second sound collector, when the monitoringtarget is out of a predetermined switching area corresponding to thefirst sound collector.
 38. The directivity control method according toclaim 21, further comprising: displaying, in a list, images captured bya plurality of imagers in different screens on the display in accordancewith a predetermined input operation; and selecting, from the pluralityof imagers, an imager for displaying an image of the monitoring targeton the display in accordance with an operation of selecting one of thedifferent screens displayed in the list on the display.
 39. Thedirectivity control method according to claim 21, further comprising:displaying markers indicating positions of a plurality of soundcollectors, which are located around the first sound collector and areswitchable from the first sound collector in accordance with apredetermined input operation; and switching collecting of the soundfrom the monitoring target from the first sound collector to a soundcollector, of the plurality of sound collectors, corresponding to aselected marker in accordance with an operation of selecting the markerfrom the markers displayed on the display.
 40. The directivity controlmethod according to claim 38, further comprising selecting, from aplurality of sound collectors including the first sound collector, asound collector for collecting the sound from the monitoring target, inaccordance with a designation of a position of the monitoring target inthe image captured by the selected imager, the selected sound collectorbeing positioned closest to the designated position of the monitoringtarget.
 41. The directivity control method according to claim 38,further comprising: detecting a direction of a face of the monitoringtarget from the image displayed on the display; and selecting, from aplurality of sound collectors including the first sound collector, asound collector for collecting the sound from the monitoring target, inaccordance with a designation of a position of the monitoring target inthe image captured by the selected imager, the selected sound collectorbeing positioned closest to the designated position of the monitoringtarget in the image captured by the selected imager in a directioncorresponding to the detected direction of the face of the monitoringtarget.
 42. The directivity control method according to claim 38,further comprising: causing a sound output device to output the soundcollected by the first sound collector; displaying, on the display,markers indicating positions of a plurality of sound collectors,including the first sound collector, corresponding to the selectedimager; sequentially outputting, for a predetermined period of time,sound having a directivity in a direction toward the monitoring targetfrom each of the plurality of sound collectors corresponding torespective one of the markers displayed on the display, in accordancewith a designation of a position of the monitoring target in the imagecaptured by the selected imager, and selecting a sound collector forcollecting the sound from the monitoring target, which corresponds to aselected marker, in accordance with an operation of selecting the markerbased on the output sound.
 43. A non-transitory computer readablestorage medium in which a program is stored, the program causing acomputer, which controls a directivity of a sound collected by a firstsound collector, including a plurality of microphones, to executeoperations comprising: displaying a captured image on a display;enlarging the captured image and displaying an enlarged image on thedisplay; forming a directivity of the sound in a direction toward amonitoring target corresponding to a first designated position in theimage displayed on the display; obtaining information on a seconddesignated position in the image displayed on the display, designated inaccordance with a movement of the monitoring target; and changing thedirectivity of the sound toward the monitoring target corresponding tothe second designated position by referring to the information on thesecond designated position.
 44. A directivity control system comprising:an imager configured to capture an image in a sound collecting area; afirst sound collector, including a plurality of microphones, configuredto collect a sound in the sound collecting area; and a directivitycontroller configured to control a directivity of the sound collected bythe first sound collector, wherein the directivity controller includes:a display controller configured to display a captured image on thedisplay, enlarge the captured image and display an enlarged image on thedisplay; a directivity former configured to form a directivity of thesound in a direction toward a monitoring target corresponding to a firstdesignated position in the image displayed on the display; and aninformation obtainer configured to obtain information on a seconddesignated position in the image displayed on the display, designated inaccordance with a movement of the monitoring target, wherein thedirectivity former changes the directivity of the sound toward themonitoring target corresponding to the second designated position byreferring to the information on the second designated position obtainedby the information obtainer.